2011 Workshop on Action, Language and...

2011 Workshop on

Action, Language and Neuroinformatics Hosted by the USC Brain Project, University of Southern California

July 25-27, 2011

Organizer: Michael A. Arbib

Database of Protocol-WrappedExperimental Results

Empirical Generalizations and Summary Tables and Figures

Models whichExplain and Predict

Pointers toAgreements

andDisagreements

Summarizationand Publication

Tools

Linksto RetrieveSupporting

Data

JournalArticles

Etc.

Venue: Hilton Checkers Hotel, 535 S. Grand Avenue | Los Angeles, CA 90071

This Workshop was supported in part by the National Science Foundation under Grant No. 0924674

(M.A. Arbib, Principal Investigator)

Website: http://neuroinformatics.usc.edu/mediawiki/index.php/Workshop

ii

Aims of the Workshop The Workshop is designed to address two linked research areas:

1) Neural mechanisms for the control, recognition and imitation of complex actions

2) Neurolinguistics, with an emphasis on embodied cognition and construction grammar informed by research on

the first topic

In doing this, we wish further to address:

3) Neuroinformatics focused on themes (1) & (2) by structuring

a) The key models from our groups and elsewhere.

b) Summaries of key data in a form which will aid the design and testing of models

with consideration of how best to develop new Collaboratory tools for the Brain Operation Database (BODB) and

federating them with other neuroinformatics resources.

We aim to discover new links between our different research programs, laying the basis for a number of new

collaborations. In addition to short presentations of research by all the participants, we will devote a full day to

structured discussions of the relations between Action, Language and Neuroinformatics. These discussions will lay

the basis for the publication of two reports:

1) Action Mechanisms in the Brain: Data, Models and Neuroinformatics

2) Language Mechanisms in the Brain: Data, Models and Neuroinformatics

Michael A. Arbib July, 2011

iii

Table of Contents Workshop Participants 1 Attendees’ Photos and Short Bios 2

Michael Arbib ...................................................................................................................................2 Lisa Aziz-Zadeh ................................................................................................................................2 Victor Barres .....................................................................................................................................3 Jason W. Bohland..............................................................................................................................3 James Bonaiuto .................................................................................................................................4 Bornkessel-Schlesewsky ...................................................................................................................4 Mihail Bota........................................................................................................................................5 Ramsay Brown ..................................................................................................................................5 Gully Burns .......................................................................................................................................6 Erica Cartmill ....................................................................................................................................6 Yiannis Demiris.................................................................................................................................6 Peter Fox ...........................................................................................................................................7 Brad Gasser .......................................................................................................................................8 Mike Grosvald...................................................................................................................................8 Madhuri Harway ...............................................................................................................................9 David Kemmerer ...............................................................................................................................9 Gerard Kempen ...............................................................................................................................10 Jinyong Lee .....................................................................................................................................11 Brian MacWhinney .........................................................................................................................11 David Marques ................................................................................................................................12 Risto Miikkulainen..........................................................................................................................12 Rick Misra .......................................................................................................................................13 Finn Årup Nielsen ...........................................................................................................................13 Erhan Oztop.....................................................................................................................................14 Matte Schilling ................................................................................................................................14 Rob Schuler .....................................................................................................................................15 Steve Small......................................................................................................................................15 Raymond Yu ...................................................................................................................................16 Wim Vanduffel................................................................................................................................16 Justin Wood.....................................................................................................................................17

Program 18 Day 1: Monday July 25 ...................................................................................................................18 Day 2: Tuesday, July 26 ..................................................................................................................18 Day 3: Wednesday July 27..............................................................................................................19

Abstracts of Presentations with Related Discussion Topics 20 Introduction .....................................................................................................................................20

Arbib: An Overview of the three themes .................................................................................20 Session 1..........................................................................................................................................20

Bonaiuto: Modeling Mirror Systems and More......................................................................20 Demiris: Modeling Imitation ....................................................................................................20 Kempen: Brain circuitry for syntax and syntactic processing in language production and

comprehension: Toward a neurocomputational model based on interactive activation and competition ......................................................................................................................21

Kemmerer: The cross-linguistic prevalence of SOV and SVO word order patterns reflects the hierarchical sequential representation of action in Broca's area.................................21

Schuler: New Developments in Collaboratory Workspaces for BODB ................................22 Session 2..........................................................................................................................................22

iv

Aziz-Zadeh: fMRI of the mirror system in action and language...........................................22 Small: Designing and Analyzing Naturalistic Biological Studies of Human Language.......22 Vanduffel: An integrated view of human and macaque data ................................................22 Bota: Revisiting the NeuroHomology Database......................................................................23 Marques: Wasn’t the web designed for data sharing? Our digital atlas and API program.23 Lee: Template Construction Grammar and the description of visual scenes.......................23

Session 3..........................................................................................................................................24 Bornkessel-Schlesewsky: Language diversity, actor-driven comprehension and

consequences for the neural language architecture.............................................................24 Barres: Viewing Template Construction Grammar as a Model of Comprehension ...........25 Grosvald: Dissociating linguistic and non-linguistic gesture processing: An ERP study of

American Sign Language.......................................................................................................26 Cartmill: From ape and human gesture to language..............................................................26 Schilling: Recruitment of Grounded Internal Models in Action, Cognition and Language26 MacWhinney: The Competition Model of language processing ............................................27 Burns: Building a Breakthrough Machine ..............................................................................28

Session 4..........................................................................................................................................28 Bohland: Informatics efforts to link gene expression data to speech/language brain systems28 Miikkulainen: Language and neural networks .......................................................................28 Oztop: From the concept of ‘self-observation for mirror neuron development’ to imitation

and robot skill transfer ..........................................................................................................29 Gasser: Ontogenetic Ritualization in nonhuman primates ....................................................29 Wood: How cognition functions in primates ...........................................................................30 Fox: BrainMap, ALE and meta-analytic modeling. ...............................................................30 Nielsen: The Brede database and new tools for meta-analysis ..............................................30

Background for the Workshop: Abstracts of Selected Papers by Participants 31 Arbib ...............................................................................................................................................31 Aziz-Zadeh ......................................................................................................................................32 Bohland ...........................................................................................................................................34 Bonaiuto ..........................................................................................................................................36 Bornkessel-Schlesewsky .................................................................................................................37 Bota .................................................................................................................................................39 Burns ...............................................................................................................................................40 Cartmill ...........................................................................................................................................41 Demiris ............................................................................................................................................43 Fox ..................................................................................................................................................44 Grosvald ..........................................................................................................................................46 Kemmerer........................................................................................................................................47 Kempen ...........................................................................................................................................49 Lee...................................................................................................................................................52 MacWhinney ...................................................................................................................................53 Miikkulainen ...................................................................................................................................55 Nielsen.............................................................................................................................................56 Oztop ...............................................................................................................................................58 Schilling ..........................................................................................................................................60 Small ...............................................................................................................................................61 Vanduffel.........................................................................................................................................63 Wood...............................................................................................................................................64

Workshop Participants

Overseas Participants

Ina Bornkessel-Schlesewsky <[email protected]> (Germany)

Yiannis Demiris <[email protected]> (England)

Gerard Kempen <[email protected]> (Netherlands) Finn Aarup Nielsen <[email protected]> (Denmark)

Erhan Oztop <[email protected]> (Japan)

Wim Vanduffel <[email protected]> (Belgium)

North American Participants

Jason Bohland <[email protected]> (Boston)

Erica Cartmill <[email protected]> (Illinois)

Peter T Fox <[email protected]> (Texas)

David Kemmerer <[email protected]> (Indiana)

Brian MacWhinney <[email protected]> (Pennsylvania)

David Marques <[email protected]> (San Francisco)

Risto Miikkulainen <[email protected]> (Texas)

Rick Misra <[email protected]>

Malte Schilling <[email protected]> (Berkeley)

Local Participants

Michael Arbib [email protected]

Lisa Aziz-Zadeh <[email protected]> (USC)

James Bonaiuto [email protected] (Caltech)

Mihail Bota <[email protected]> (USC)

Gully Burns <[email protected]> (USC/ISI)

Mike Grosvald <[email protected]> (Irvine)

Steven L. Small [email protected] (Irvine)

Justin Wood <[email protected]> (USC)

Arbib’s Students

Jinyong Lee <[email protected]>

Robert Schuler <[email protected]>

Brad Gasser <[email protected]>

Victor Barres <[email protected]>

Madhuri Harway [email protected]

Other Students

Ramsay Brown [email protected]

Raymond Yu [email protected] (National University of Taiwan)

2

Attendees’ Photos and Short Bios

Michael Arbib

Michael A. Arbib is the Fletcher Jones Professor of Computer Science, as well as a Professor of Biological

Sciences, Biomedical Engineering, Electrical Engineering, Neuroscience and Psychology at USC. The thrust of his

work is expressed in the title of his first book, Brains, Machines and Mathematics (McGraw-Hill, 1964). The brain

is not a computer in the current technological sense, but he has based his career on the argument that we can learn

much about machines from studying brains, and much about brains from studying machines. His research has long

included a focus on brain mechanisms underlying the coordination of perception and action. His group prepared the

first computational model of mirror neurons and conducted some of the key initial imaging studies of the human

mirror system. He is now developing a new theory of the evolution of human language. He has also contributed to

neuroinformatics, with the current emphasis on BODB, the Brain Operation Database.

Lisa Aziz-Zadeh

Lisa Aziz-Zadeh's expertise is in cognitive neuroscience, with specific training in functional neuroimaging, the

mirror neuron system, and embodied semantics and cognition. As a graduate student at UCLA, she conducted

several fMRI and TMS studies on the mirror system, laterality, and language. Her training continued during her

post-doctoral fellowship at the University of Parma, Italy, where she utilized fMRI to study embodied semantics for

language as well as explored both covert and overt speech. She completed a second post-doc at UC Berkeley, where

together with Richard Ivry, Srini Naranayan and Jerome Feldman, she continued brain imaging work on embodied

semantics, supporting the previous finding to regions outside the motor cortex. In her current position at USC in the

Brain and Creativity Institute and the Division of Occupational Science and Occupational Therapy, Lisa has

3

continued exploring the neural basis of language processing using fMRI, in particular investigating how sensory

regions contribute to linguistic processing as well as other components of social cognition and emotion processing.

Victor Barres

Victor Barres is a PhD candidate in the neuroscience department at the University of Southern California under

the direction of Dr. Arbib. He received his M.S. degree in quantum physics from the Ecole Polytechnique, France in

2006, and his M.S degree in cognitive science from Ecole Normale Superieure, Paris in 2010. He focused his PhD

work on neurolinguistics and is currently working on a parsing model extending the SemRep/TCG system

developed by Lee and Arbib. His interests include computational models of cognitive systems, the role of motor and

spatial schemas in building semantic representations, and language emergence.

Jason W. Bohland

Jason W. Bohland is an Assistant Professor in the Department of Health Sciences at Boston University as well

as a faculty member in the cross-departmental Graduate Program for Neuroscience. I received my PhD in Cognitive

and Neural Systems from Boston University in 2007 with Dr. Frank Guenther, studying speech motor planning

using computational and neuroimaging techniques. I went on to a postdoctoral fellowship with Dr. Partha Mitra at

Cold Spring Harbor Laboratory, working generally in the area of neuroinformatics, with emphasis on understanding

large scale properties of the molecular and connectivity architecture of the brain, and helping to launch the Mouse

Brain Architecture Project, which aims to systematically map projection patterns in wild type and disease model

mice. At Boston University my laboratory brings together these distinct elements of my experience as we combine

computational, informatics, and neuroimaging methods to elucidate the structural and functional organization of

brain systems that support speech and language systems.

4

James Bonaiuto

James “Jimmy” Bonaiuto is a Swartz Postdoctoral Fellow at the California Institute of Technology. He received

a B.S. in Computer Science from Drexel University in 2004 and a Ph.D. in Neuroscience at the University of

Southern California in 2010. His doctoral thesis was completed under the advisement of Michael Arbib and was

entitled, “Modeling the Mirror System in Action Observation and Execution.” His research interests include

computational modeling of reaching and grasping, action observation, and imitation, synthetic brain imaging, and

neuroinformatics. Since 2010 he has been working as a postdoctoral researcher in Richard Andersen’s lab on fMRI

of spatial decision-making in awake, behaving monkeys.

Bornkessel-Schlesewsky

Ina Bornkessel-Schlesewsky is Professor of Neurolinguistics at the University of Marburg. She studied General

Linguistics and Computational Linguistics at the University of Potsdam and received her PhD from there in 2002.

From 2005 to 2010 she was the head of the research group "Neurotypology" at the Max Planck Institute for Human

Cognitive and Brain Sciences in Leipzig. In her research, she studies the neural bases of language comprehension,

with a particular focus on the implications of cross-linguistic diversity. To this end, she co-founded, together with

Matthias Schlesewsky, the new research field of neurotypology, which seeks to combine insights from

neurolinguistics and language typology in order to shed new light on the language architecture and how it is

implemented by the brain. A crucial part of this work has lain in the development of a cross-linguistically adequate

neurolinguistic model of language comprehension, the extended Argument Dependency Model (eADM). Current

research interests including broadening the scope of the eADM's coverage to include the discourse level and

grammatical relations, predictions for neuroanatomy and the use of naturalistic stimuli.

5

Mihail Bota

Mihail Bota is an Associate (Research) Professor in the Department of Neurobiology, University of Southern

California. His former education include Physics (Biophysics) and Philosophy. His Ph.D. Thesis was a study of the

neural homologies of the rat, monkey and human cortex, with application to the premotor cortex . He was the

developer and main collator and curator of the NeuroHomology Database, under the supervision of Professor

Michael Arbib. His more recent work includes development and population of the Brain Architecture Management

System (BAMS; http://brancusi.usc.edu/bkms; http://brancusi1.usc.edu), which is an online publicly accessible

neuroinformatic workbench that handles data across several levels of the mammalian central nervous system. His

research interests include construction of functionally relevant neural circuits from data collated from the literature,

connectome construction and comparison of neural features across different mammalian species, and construction of

controlled vocabularies (ontologies) as standards for interoperability across different neuroinformatic platforms.

Ramsay Brown

Ramsay Brown is a graduate student at the University of Southern California. He earned his bachelors at USC in

Neuroscience, and has been admitted to the Neurobiology PhD Program starting this fall. Working with Dr. Larry

Swanson and Dr. Mihail Bota, Ramsay is actively developing the "flatmap" online graphical user interface to the

Brain Architecture Management System and looks forward to its launch this coming year. He is interested in how

neural networks found in the brain underlie goal-oriented survival behavior, the exploration of network structure

through experimental neuroanatomy, and what network topology can tell us about network function.

6

Gully Burns

Gully received an undergraduate degree in Physics from Imperial College in London, and a Doctoral degree in

Physiology from Oxford University. He is interested in developing Knowledge Engineering methods for Bio-

medicine that allow scientists to reason and process more knowledge. His main research interests are to understand

and develop systems that enable and even encourage scientists to make breakthroughs.

Erica Cartmill

Erica Cartmill is a postdoctoral scholar in the Department of Psychology at the University of Chicago. She works

with Professor Susan Goldin-Meadow on the role that gesture plays in language acquisition and in grounding speech

in the physical world. Erica earned her PhD in 2009 from the University of St Andrews under the supervision of

Professor Richard Byrne for her research on orangutan gestural communication. Her work with both children and

apes aims to understand environmental influences on communication/language and the role gesture plays in the

development of communication systems on both evolutionary and ontogenetic timescales.

Yiannis Demiris

7

Yiannis Demiris heads the Bioinspired Assistive Robots and Teams (BioART) laboratory at the Department of

Electrical and Electronic Engineering of Imperial College London, doing research on human-robot interaction,

assistive robotics, and multirobot teams, with an emphasis on biologically inspired mechanisms of social

development and social learning. He received his PhD in Intelligent Robotics from the Department of Artificial

Intelligence of the University of Edinburgh in 1999, and joined the faculty of Imperial College in 2001. His current

projects include two European FP7 projects, ALIZ-E (Adaptive Strategies for Sustainable Long-Term interaction)

and EFAA (Experimental Functional Android Assistant) where he is aiming to develop bio-inspired mirror and

imitation systems for enabling robots to perceive, learn from and assist human actions in multimodal tasks, ranging

from classical assembly tasks to musical instrument playing and dance with adult and children subjects.

Peter Fox

Dr. Fox is the Director of the Research Imaging Institute (RII; http://rii.uthscsa.edu/) of the University of Texas

Health Science Center at San Antonio (UTHSCSA). His primary academic appointment is in the Department of

Radiology, in which he holds the Malcolm Jones Chair and serves as the Vice Chair of Research and Research

Education. He holds joint appointments in Neurology, Psychiatry and Physiology and holds research appointments

at the South Texas Veterans Health Care System and the Southwest National Primate Center. Hehas been engaged in

neuroimaging research for over 30 years, starting his career at the Radiological Sciences Division of the

Mallinckrodt Institute at Washington University, St. Louis. Dr. Fox pioneered the use of standardized coordinates

for reporting neuroimaging results and created the BrainMap database (www.brainmap.org), which is a resource for

meta-analysis and modeling of functional and structural neuroimaging results. He received the UTHSCSA

Presidential Research Scholar Award, is a Fellow of the AAAS and has been recognized by ISI as one of the most

highly cited neuroscientists. Current interests include: modeling neural systems from neuroimaging data; meta-

analysis methods; optimal strategies for assessing treatment effects in neurological and psychiatric disorders;

hemodynamic and metabolic physiology and pathophysiology.

8

Brad Gasser

Brad Gasser is a Ph.D. student in the Neuroscience Graduate Program at the University of Southern California. I

am working with Dr Michael Arbib on building computational models of primate brain systems involved in

behavioral control and social learning. In particular, we are concerned with the functional brain changes in the

hominid line that would lead to the 'language-ready brain' in homo sapiens. I received degrees in philosophy and

psychology at Bowling Green State University in Ohio, where I worked with Dr Verner Bingman recording

hippocampal neurons in pigeons. I was also an NSF-sponsored intern with Dr David Yager at the University of

Maryland, College Park studying praying mantid escape behavior in flight.

Mike Grosvald

Mike Grosvald completed his MA in Mathematics at UC Berkeley and his PhD in Linguistics at UC Davis; his

dissertation was a production and perception study of coarticulation in spoken and signed language. As a PhD

student working in David Corina's lab, he also worked on a number of other projects, using both behavioral and ERP

methodology, investigating how signs and other human actions are processed by signing and sign-naive individuals.

In 2010, he began working in the Solodkin/Small lab, where he is using behavioral and fMRI methodology to

explore issues related to audio-visual integration (at the word level), as well as semantic and syntactic processing (at

the sentence level).

9

Madhuri Harway

Madhuri Harway is doing her Master's degree in Electrical Engineering at USC. Inspired by Richard Byrne's

work on imitation by behavioral parsing in young gorillas, she has made a set of videos of people making tea in the

Indian style, and is annotating these as the basis for assessing learning methods that could perform behavioral

parsing on widely different performances of the same skill.

David Kemmerer

I have a joint appointment in the Department of Speech, Language, and Hearing Sciences and the Department of

Psychological Sciences at Purdue University. My teaching responsibilities include courses on the neural bases of

speech and language, the field of cognitive neuroscience, and topics in linguistics. My research focuses on how

different kinds of linguistic meaning are mediated by different neural systems, drawing on behavioral and lesion

data from brain-damaged patients as well as behavioral and functional neuroimaging data from healthy subjects. My

current projects include the linguistic encoding of action and the syntax-semantics interface. I also like to follow

literature on human evolution and on various non-linguistic topics in cognitive neuroscience, such as the neural

correlates of consciousness. Mostly, though, I've been consumed with writing a textbook called "The cognitive

neuroscience of language: An introduction," under contract with Psychology Press. My hobbies include road biking,

cooking, exploring the back roads of Indiana, and reading novels, short stories, and "popular science" books.

10

Gerard Kempen

Gerard A.M. Kempen (1943) has been professor of cognitive psychology at Leiden University since 1992

(emeritus since 2008), and Research Associate of the Max Planck Institute for Psycholinguistics in Nijmegen since

1999. From 1976 to 1992 he was professor of psycholinguistics at the Radboud University in Nijmegen, where he

had received his PhD in 1970.

His psycholinguistic work concerns the grammatical aspects of human sentence processing during language

production and comprehension. He is studying these processes through a combination of experimental-psychological,

linguistic, and computational methods. His contributions to this topic include:

• the division of labor and the interaction between conceptualization and formulation processes in sentence

production (1977)

• the concept of incrementality in language production (1982)

• the distinction between lemmas and lexemes in the Mental Lexicon for language production (1983)

• an early computational model of human sentence production (1987)

• the (neuro)cognitive model of parsing called Unification Space (1989-2009)

• the (neuro)cognitively motivated Performance Grammar formalism (1991-2003)

• a grammatical theory of clausal coordinate ellipsis based on similarities with speech error repairs (2009)

• experimental and theoretical arguments for large-scale overlap between the grammatical processing

mechanisms underlying sentence comprehension and those underlying sentence production (2000-2011)

• a new model of self-monitoring for grammatical and lexical speech errors, which does not require the

“perceptual loop” (2011).

Since 1980 he initiated and supervised various theoretical and applied research projects dealing with the

computational treatment of Dutch, among other things, for visual-interactive teaching of grammatical structures

(sentence analysis) in secondary education.

11

Jinyong Lee

Jinyong Lee is a Ph.D. student in the Department of Computer Science of the University of Southern California.

He received his B.S. degree in Electrical and Computer Engineering in 2005 from Hanyang University in Seoul,

Korea and the M.S. degree in Computer Science from the University of Southern California in 2007. His research

interests include modeling human language production and comprehension, semantic representation,

neurolinguistics and psycholinguistics. He is currently working on a model of production of scene description. This

work involves building a computational model (Template Construction Grammar and its extension) with a new type

of semantic representation, SemRep, as well as conducting a set of eye-tracking experiments to collect supporting

evidence for the model.

Brian MacWhinney

Brian MacWhinney is Professor of Psychology, Computational Linguistics, and Modern Languages at Carnegie

Mellon University. He received a Ph.D. in psycholinguistics in 1974 from the University of California at Berkeley.

He has developed a model of first and second language processing and acquisition based on competition between

item-based patterns. In 1984, he and Catherine Snow co-founded the CHILDES (Child Language Data Exchange

System) Project for the computational study of child language transcript data. The CHILDES programs and database

have now become an important component of the basic methodology of research in language acquisition. He is now

extending this system to six additional research areas in the form of the TalkBank Project. MacWhinney’s recent

work includes studies of online learning of second language vocabulary and grammar, neural network modeling of

lexical development, fMRI studies of children with focal brain lesions, and ERP studies of between-language

competition. He is also exploring the role of grammatical constructions in the marking of perspective shifting, the

determination of linguistic forms across contrasting time frames, and the construction of mental models in scientific

reasoning.

12

David Marques

David Marques is currently VP of Business Development for the Books group in the Science and Technology

division of Elsevier. He has been at Elsevier since 1999, when he took the job of Chief Technology Officer and

created and managed Elsevier Labs and Elsevier User-Centered Design. David holds an A.B. degree in Psychology

from Cornell University (1972), a Ph.D. in Psychobiology from Michigan (1976), and did post-doctoral research in

neuroscience at Pittsburgh and at the Worcester Foundation for Experimental Biology. In 1981, David took a left

turn into technology, working for 15 years at Digital Equipment Corporation in software consulting, Artificial

Intelligence Research, Education market technology, and collaboration software at AltaVista. His most recent work

has been to lead the product development of Elsevier’s BrainNavigator.

Risto Miikkulainen

Risto Miikkulainen is a Professor of Computer Sciences at the University of Texas at Austin. He received an

M.S. in Engineering from the Helsinki University of Technology, Finland, in 1986, and a Ph.D. in Computer

Science from UCLA in 1990. His current research includes models of natural language processing, self-organization

of the visual cortex, and evolving neural networks with genetic algorithms; he is an author of over 250 articles in

these research areas. He is currently on the Board of Governors of the Neural Network Society, and an action editor

of IEEE Transactions on Autonomous Mental Development, IEEE Transactions on Computational Intelligence and

AI in Games, the Machine Learning Journal, Journal of Cognitive Systems Research, and Neural Networks.

13

Rick Misra

Kaushik (Rick) Misra is a member of Elsevier’s BrainNavigator product development team and a collaborating

scientist at the Scripps Research Institute. His Ph.D. involved investigating the molecular and genetic mechanisms

of alcohol addiction and mood disorders. His first post-doctoral work used siRNA, antisense, and pharmacological

blockade techniques to examine the interaction between neuropeptide and neurotrophic systems. He later took a

post-doctoral position at the Scripps Research Institute using behavioral, molecular, and pharmacological techniques

as a means of examining the involvement of neuropeptides associated with pain and stress systems in opiate

dependence. He began work in the field of neuroinformatics and neuroscience based analytic tools as a product

manager for Elsevier’s BrainNavigator. His particular focus remains creating an interface that utilizes neuroanatomy

as a platform for structure based neuroinformatics.

Finn Årup Nielsen

Finn Årup Nielsen is Senior Researcher at the Technical University of Denmark, where he also completed his

PhD. His dissertation was entitled "Neuroinformatics in Functional Neuroimaging" and he has since been working

on combining scientific and web information both in the form of numerical data, text and ontology components. He

has developed two Matlab-based toolboxes: One ("Lyngby") for primarily analyzing original neuroimaging data in a

variety of ways, another ("Brede") for alternative analysis with text mining and meta-analysis and handling of Web

data. Initially working with the BrainMap database, he developed the Brede Database and now focuses on a wiki-

oriented approach with the Brede Wiki for storing, quering and analyzing numerical data from scientific studies.

14

Erhan Oztop

Erhan Oztop earned his Ph.D. at the University of Southern California in 2002. In the same year, he joined the

Computational Neuroscience Laboratories at Advanced Telecommunications Research Institute International (ATR)

in Japan where he worked as a researcher until 2004. Immediately after, he joined the JST ICORP Computational

Project as a researcher, and later became a group leader (2004-2008). Currently he works at NICT, Advanced ICT

Research Institute, Brain ICT Laboratory and is a senior researcher of ATR Cognitive Mechanisms Laboratories

where he is leading the Communication and Cognitive Cybernetics group. He also holds a visiting associate

professor position at Osaka University. His research interests include computational and cognitive neuroscience,

human-robot interaction and computational modeling and analysis of intelligent behavior.

Matte Schilling

Malte Schilling is a PostDoc at the International Computer Science Institute (ICSI) in Berkeley. His work

concentrates on internal models, their grounding in behaviour and their application in higher-level cognitive function

like planning ahead or communication.

Before moving to Berkeley, he worked at the Sony Computer Science Laboratory in Paris on the embodiment of

language and the connections between internal models and language.

He received his PhD in Biology at the University of Bielefeld in January, 2010. His PhD project focuses on the

control of a hexapod robot, applying insights from biological experiments with stick insects on a reactive control

model, and enhancing this control through cognitive capabilities by introducing an internal body model implemented

as a neuronal network. After finishing his PhD he became a Responsible Investigator at the Center of Excellence

'Cognitive Interaction Technology' (CITEC), University of Bielefeld.

15

He has studied Computer Science (artificial intelligence as a main subject) from 1997-2003 at the University of

Bielefeld and finished the Diploma with his thesis on knowledgebased systems in the context of virtual

environments.

Rob Schuler

Rob Schuler is a Ph.D. candidate in the Dept. of Computer Science of the University of Southern California. He

received the B.S. degree in Computer Science in 1995 and the M.S. degree in Computer Science in 2008, both from

USC. His research interests include scientific data management, scientific collaboratories, and databases and tools to

support neuroscience. Since 2004, Rob has worked as a systems programmer at the USC Information Sciences

Institute working on large-scale data management systems for biomedical, physical, and climate sciences. Most

recently, he has served as a member of the steering committee of the Biomedical Informatics Research Network

(BIRN). Previously, he held a variety of positions in the technology industry including trusted systems development

for a division of Xerox Research & Technology.

Steve Small

Steven L. Small is the Stanley van den Noort Professor and Chair of Neurology, as well as Professor of

Neurobiology and Behavior and Cognitive Sciences at the University of California, Irvine. He is also Professor

Emeritus of Neurology and Psychology at The University of Chicago, and serves as Editor in Chief of the

international journal Brain and Language. The main focus of his efforts over the past two decades has been in the

neurobiology of language, and he has recently founded an international organization dedicated to this topic, the

Society for the Neurobiology of Language, and an associated conference, the Neurobiology of Language Conference,

which recently had its second annual meeting with over 400 participants. Why a new society and conference? The

emphasis of work from this new perspective is to uncover the neural mechanisms of language, i.e., the specific

implementations of language functions in neurons, axons, glial cells, electrical signals, chemical messengers, genetic

16

expression, and related biological entities. This differs from the other approaches to the study of language that use

brain measurements, in which the focus is on cognitive systems, linguistic theories, and other non-biological

explanations. In Small's laboratory, research has primarily used functional MRI and has investigated mechanisms of

speech perception and naturalistic language comprehension, with a recent emphasis on the extent to which sensory

and motor neural codes are involved in everyday language use.

Raymond Yu

Raymond Yu is a Ph.D. candidate in the Department of Computer Science, Nation Taiwan University, Taipei,

Taiwan. His research interests are in constructing computational models which help to understand the function of the

brain, especially functions related to the human visual process. Currently he is working on a computational model

involving prefronto-parietal interactions for the cognitive functions of visual-spatial sketchpad, counting, and scene

integration. This work was initiated during the calendar year 2010 when he was a Visitor in Michael Arbib's lab at

USC.

Wim Vanduffel

Wim Vanduffel conducts research at the Lab for Neuro-and Psychophysiology, K.U.Leuven, Belgium, as well as

being Assistant in Neuroscience at MGH and Assistant Professor in Radiology, Harvard Medical School. He has

extensive expertise in the use of neuroimaging to study humans and awake behaving monkeys. His specific area of

interest is the primate visual system. He compares directly the functional organization of the visual system in human

and non-human primates using the fMRI technique and novel comparative methods. He currently focuses on the

functional role of feedback connections within the attention system by performing ‘perturb-and-measurement’

techniques in non-human primates. In particular, he investigates the behavioral and functional consequences of

17

reversible (in)activations of a node within a cortical network. He also studies the effect of attention and reward

signals on sensory processing.

Justin Wood

Assistant Professor of Psychology, College of Letters, Arts, & Sciences. Educational Background: University of

Virginia, Harvard University. Research examines the origins of the mind in relation to how knowledge emerges

during development and may have emerged on a biological scale. Studies a wide range of cognitive abilities,

including mathematics, object perception and cognition, and the social systems that allow humans and nonhuman

animals to make inferences about others' goals, intentions, beliefs, and desires Recipient of New Investigator Award

from the American Psychological Association

18

Program

All sessions will be held in Room A except that breakout sessions will have their separate rooms on Wednesday.

Coffee will be available from 8:30 in Room A on Monday and Tuesday and in the breakout rooms on

Wednesday.

The banquet will be held on the Checkers Patio, with a Reception at 7 and seating for dinner at 7:30.

Day 1: Monday July 25

9:00-10:00: Arbib – An Overview of the three themes

10:00-1:00: Session 1: Five 30 minute talks: [Coffee break 10:30-11:00]

Bonaiuto: Modeling Mirror Systems and More

Demiris: Modeling Imitation

Kempen: Brain circuitry for syntax and syntactic processing in language production and comprehension:

Toward a neurocomputational model based on interactive activation and competition

Kemmerer: The cross-linguistic prevalence of SOV and SVO word order patterns reflects the hierarchical

sequential representation of action in Broca's area

Schuler: New Developments in Collaboratory Workspaces for BODB

1:00-2:00: Lunch

2:00-5:30: Session 2: Six 30 minute talks [Coffee break 3:30-4:00]

Aziz-Zadeh: fMRI of the mirror system in action and language

Schilling: Recruitment of Grounded Internal Models in Action, Cognition and Language

Vanduffel: An integrated view of human and macaque data

Bota: Revisiting the NeuroHomology Database

Marques: Wasn’t the web designed for data sharing? Our digital atlas and API program

Lee: Template Construction Grammar and the description of visual scenes

Day 2: Tuesday, July 26

9:00-1:00: Session 3: Seven 30 minute talks [Coffee break 10:30-11:00]

Bornkessel-Schlesewsky: Language diversity, actor-driven comprehension and consequences for the

neural language architecture

Barres: Viewing Template Construction Grammar as a Model of Comprehension

Grosvald: Dissociating linguistic and non-linguistic gesture processing: An ERP study of American Sign

Language

Cartmill: From ape and human gesture to language

Small: Designing and Analyzing Naturalistic Biological Studies of Human Language

MacWhinney: The Competition Model of language processing

Burns: Building a Breakthrough Machine

1:00-2:00: Lunch

2:00-6:00: Session 4: Seven 30 minute talks: [Coffee break 3:30-4:00]

Bohland: Informatics efforts to link gene expression data to speech/language brain systems

19

Miikkulainen: Language and neural networks

Oztop: From the concept of ‘self-observation for mirror neuron development’ to imitation and robot skill

transfer

Gasser: Ontogenetic Ritualization in nonhuman primates Wood: How cognition functions in primates

Fox: BrainMap, ALE and meta-analytic modeling

Nielsen: The Brede database and new tools for meta-analysis

6:00-7:00 Free time

7:00Reception

7:30: Banquet

Day 3: Wednesday July 27

9:00-11:30: Three parallel Breakout Sessions 1: Cross-cutting groups. [Refreshments in meeting rooms.]

[The name of the rapporteur for each session is marked in Bold.]

1a. Half the Action group + half the Neuroinformatics group: assessing the model-data integration and

neuroinformatics needs of the Action group: Bonaiuto, Oztop, Demiris, Vanduffel, Arbib, Marques, Bohland, Bota

1b. Half the Neurolinguistics group + the other half of the Neuroinformatics group: assessing the model-data

integration and neuroinformatics needs of the Neurolinguistics group: Bornkessel-Schlesewsky, Small,

MacWhinney, Miikkulainen, Nielsen, Fox, Barres, Schuler

1c. The other half of the Action group + the other half of the Neurolinguistics group: Defining shared modeling

challenges and the development of a shared conceptual framework. Kemmerer, Aziz-Zadeh, Cartmill, Gasser,

Grosvald, Wood, Kempen, Lee, Schilling.

11:30-12:30: Lunch

12:30-2:30: Three parallel Breakout Sessions 2: One group for each theme meet separately.

2a. Action group: What are the key data and/or conceptual issues ripe for modeling; what are the key lines of

modeling that hold most promise to address these data/issues? Oztop, Demiris, Vanduffel, Cartmill, Arbib, Aziz-

Zadeh, Gasser, Schilling, Wood

2b. Neurolinguistics group: What are the key data and/or conceptual issues ripe for modeling; what are the key

lines of modeling that hold most promise to address these data/issues? MacWhinney, Kempen, Grosvald, Small,

Bornkessel-Schlesewsky, Miikkulainen. Kemmerer, Lee, Barres

2c. Neuroinformatics group: What tools are ripe for sharing, or should be ripened? What are promising lines for

federation? Nielsen, Marques, Bohland, Bonaiuto, Bota, Fox, Schuler

2:30-3:00: Coffee break.

3:00-5:00: Concluding Plenary Session: Each rapporteur will present a 10 minute report on the findings of his/her

group followed by 10 minutes of general discussion. These reports will lay the grounds for future research and

collaboration.

5:00 Adjourn

20

Abstracts of Presentations with Related Discussion Topics

Introduction

Arbib: An Overview of the three themes This will provide a brief introduction to our work on modeling neural mechanisms for action and action

recognition; an evolutionary scenario linking action, action recognition, imitation and language; and our Brain

Operation Database, BODB. Synthetic Brain Imaging (SBI) will be introduced as a way to integrate modeling based

on data from the macaque and human brain. The presentation will provide a framework for the following

presentations and discussion sessions.

Session 1

Bonaiuto: Modeling Mirror Systems and More Both premotor and parietal cortex of the macaque brain contain mirror neurons each of which fires vigorously

both when the monkey executes a certain limited set of actions and when the monkey observes some other perform a

similar action. Turning to the human, we must rely on brain imaging rather than single-neuron recording. I will

present biologically plausible models of the mirror system and its interactions with other brain regions in grasp

observation and execution. The Mirror Neuron System (MNS2) model suggests an extension to the original model

that addresses recent data on the response of mirror neurons to sounds characteristically associated with an action,

and to grasps in which the final portion is occluded by an obstruction. The Integrated Learning of Grasps and

Affordances (ILGA) model gives an account of how F5 canonical and AIP visual neurons, which provide input to

the MNS2 model, simultaneously gain their properties through development. Augmented Competitive Queuing

(ACQ) then shows how systems like ILGA and MNS2 can be deployed and used to chain together sequences of

actions and rapidly reorganize these sequences in the face of disruption. Finally I will briefly discuss the use of

synthetic brain imaging to allow computational models to address monkey and human data.

Additional Discussion Points for Wednesday

Extending the above models to models of imitation.

Further development of SBI, and links to work of Aziz-Zadeh and Vanduffel .

Challenges for BODB development.

Demiris: Modeling Imitation For the last few years, I have been working on modeling mechanisms of social learning and imitation trying to

simultaneously stay close to biological data as well as implementing the mechanisms on robotic systems. Starting

from data on imitation in adults and infants, data on visuo-imitative apraxia, as well as gaze tracking during human

action perception, I proposed a model that involves a dual-route process for imitation and understanding actions,

involving both passive (learning) and active (mirror-like) computational elements. Instantiation of the model over

several years on a variety of tasks (on robots, adversarial computer games and assistive robotic wheelchairs)

revealed a number of lessons for modellers including the need for perspective taking, hierarchical processing, and

principled control of attention during action perception, and put forward a number of testable predictions regarding

the behavior of mirror neurons.

21


Linking to Oztop and Gasser: How can this model inform computational neuroscience?

Role of working memory and attention

Positive transfer to structurally similar tasks: cf. Wood & Cartmill

Perspective taking

Computational strategies for social learning and imitation; hierarchical representations for behavioral control and

strategies for learning them.

Kempen: Brain circuitry for syntax and syntactic processing in language production and

comprehension: Toward a neurocomputational model based on interactive activation and

competition Space model of grammatical decoding (parsing). Presentation of new behavioral and fMRI evidence in favor of

the idea that grammatical decoding in sentence comprehension utilizes the same neurocognitive infrastructure as

grammatical encoding in sentence production. Discussion of implications for the design of a direction-neutral

neuronal mechanism for grammatical decoding *and* encoding.


Compare/contrast Performance Grammar and Construction Grammar.

Could direction-neutral grammatical encoding/decoding involve mirror neurons?

Plausibility of self-organizing or optimization mechanisms other than interactive activation and competition as

models of syntactic processing.

Response to Bornkessel-Schlesewsky critique of 2006.

Relation to MacWhinney competition model.

Kemmerer: The cross-linguistic prevalence of SOV and SVO word order patterns reflects the

hierarchical sequential representation of action in Broca's area It is well-established that the vast majority of the roughly 7,000 languages in the world have either SOV (about

48%) or SVO (about 41%) word order (Dryer, 2011). Several scholars have argued that these strong tendencies can

be explained cognitively in terms of the prototypical transitive action scenario, in which an animate agent acts

forcefully on an inanimate patient and thereby induces a change of state. Two forms of iconicity are especially

relevant: first, because the agent is at the head of the causal chain that affects the patient, subjects usually precede

objects; and second, because it is the agent’s action, rather than the agent per se, that changes the state of the patient,

verbs and objects are usually adjacent. My aim is to show that this account can be deepened and enriched by relating

it to recent research on how actions are represented in the brain. Specifically, I review several lines of evidence

which, taken together, suggest that Broca's area—more precisely, BA44—plays a pivotal role in representing the

hierarchical sequential organization of goal-oriented bodily movements, not only when they are performed and

perceived, but also when they are linguistically described. Based on these findings, I propose that the most cross-

linguistically prevalent word order patterns reflect the most natural ways of selecting and nesting action components

in BA44.


22

Relations to Aziz-Zadeh and Vanduffel on the MNS.

Relation to Bornkessel-Schlesewsky's work on word order

Relation to MacWhinney's work on starting points and perspectives

Relation to Lee’s linkage of vision and language, and the implication of different styles of action and related

verbs.

Schuler: New Developments in Collaboratory Workspaces for BODB The talk will assume familiarity with basic design ideas of BODB (e.g., BOPs, SEDs, SSRs, and present current

work on developing Workspace resources to promote collaborations between modelers and experimentalists on

specific topics in developing summaries of related empirical data and building models which address them.


What Workspaces could be developed for different subsets of participants at the Workshop to foster future

collaboration?

How might BODB best be federated with the other databases discussed at the Workshop so that their data can be

deployed to help efforts in modeling?

Session 2

Aziz-Zadeh: fMRI of the mirror system in action and language The Human Mirror Neuron System, Embodied Representation & Embodied Semantics; relation to the

'mentalizing' systems in humans


Relating human data to insights from monkey studies.

Relation of MNS and mentalizing systems to (medial) cognitive control networks

What repair and rehabilitation can tell us about large scale brain networks

Challenges of linking relevant data sets from fMRI, TMS, and neuroanatomy.

Small: Designing and Analyzing Naturalistic Biological Studies of Human Language Addressing the use of fMRI in investigations of the neurobiology of language. This requires a substantial

reworking of design and analysis methods for fMRI, and modeling the peaks and valleys in raw fMRI time series is

a valuable approach.

Additional Discussion Points for Wednesday The relation between action understanding and language comprehension.

The relation of fMRI and ERP data.

Vanduffel: An integrated view of human and macaque data Recent models predicted that human-specific cognitive abilities emerged from recycled neuronal circuits. Instead,

evolutionary models based on cortical expansion suggested that human-specific functions are carried by novel

associative regions, without correspondent areas in other species. Yet no direct experimental evidence exists to

support either hypothesis. Using novel comparative tools which I will discuss in detail, we revealed spatially

correspondent brain networks, such as the language network, with different functional processing as predicted by the

recycling theory. In regions with larger cortical expansion, however, we observed human-specific networks with

23

completely distinct functional signatures compared to the macaque. Importantly, their presumed functions suggest a

direct link with human-specific cognitive abilities, such as abstract reasoning. Thus both neuronal recycling and

cortical expansion account for the emergence human-specific cognitive abilities.


Towards model-free methods for comparative functional studies

How to decide that a particular cognitive function is human-specific?

Are 'human-specific' cognitive functions carried by novel brain regions or by recycled brain regions?

Issues specific and common to neuroinformatics with monkey and human imaging data

Link to Aziz-Zadeh, relating MNS to other regions; Bota on neo NHDB

Bota: Revisiting the NeuroHomology Database This talk will be composed of three parts. First, I will review the NeuroHomology Database, its scope, addressed

challenges, and the results obtained by data mining. The second part will be a review of the latest development of

BAMS. Finally, I will address the possible points of convergence of these two systems, towards creation of a new

neuroinformatic system, neoNHDB.


Vanduffel as a customer of neoNHDB

Challenges of collating data for action, action recognition in macaque and human and relating them to

neurolinguistic data.

Progress and possibilities for federating BAMS with other resources.

Marques: Wasn’t the web designed for data sharing? Our digital atlas and API program. In the past 2 years, we have put 4 stereotaxic atlases online, organized by a new ontogeny-defined brain structure

ontology (from Charles Watson, Luis Puelles, George Paxinos) and toolsets to allow comparison with researcher

histology and to help visualize structures and fiber tracts in 3D to facilitate intervention planning and interpretation

accuracy. Toward our vision of helping bring together structure and function information to enhance understanding,

we link out to structure-specific data sources such as BODB and BAMS, provide APIs for others to retrieve

structure and atlas information, and are building public MRI templates to help co-register subjects to the reference

atlases. This paper presents details of how the models were constructed and the APIs that are available, and poses

the question: What APIs would be of value, where should this vision go? Additional Discussion Points for Wednesday

Work with Bonaiuto on the Macaque Brain Device.

Linking across different atlases and species

Suggestions for encouraging adoption of BODB

Link via Allen Brain Atlas to Bohland.

Lee: Template Construction Grammar and the description of visual scenes A computational model of production of descriptions of visual scenes was previously outlined by Arbib & Lee

(2008). One part of this effort was the development of a new kind of semantic representation, SemRep. SemRep is

an abstract form of visual information with an emphasis on the spatial linkage of entities, attributes and actions.

SemRep provides a graph-like hierarchical structure with enough formal semantics for verbal description of a scene.

24

Moreover, Construction Grammar was adopted as an appropriate framework for a schema-based linguistics, and

Template Construction Grammar (TCG) has been implemented as an operational version of such a framework.

Constructions, represented as schema instances, compete and cooperate to cover the SemRep to produce a

description of the visual scene at hand. The present study reports on a language model which expands the previous

account of SemRep and TCG by providing a particular mechanism of working memory (WM) and attentional focus.

The proposed mechanism, which is closely related to Cowan’s (1999) WM model, blends the focus of attention with

the WM capacity by having what is being attended and what is stored in the WM identical. In this approach, the

vision system interprets a part of the scene under the current focus of attention, and then the SemRep is created or

updated accordingly in the WM. The language system applies constructions on the SemRep built in the WM

according to the TCG rule. Therefore, the deployment mechanism of attentional focus is hypothesized to be the main

driving force of the resultant sentential structure being produced. The notion of subscene, whose preliminary

definition was provided previously through the account of the minimal subscene (Itti & Arbib, 2006), is proposed as

the unit of attentional process. A subscene is an area covered by entities of the scene under the current attentional

focus, which comes with various spanning coverage from parts of an entity to the entire scene. SemReps

corresponding to subscenes with different attentional spans are hierarchically organized and processed by the system

as attentional focus plays a role as an executive window while travelling across the hierarchy by zooming in/out and

shifting. The WM capacity limits the number (and quality) of entities that are simultaneously captured by the

attentional focus and processed in WM while the structure of the produced sentences is affected by a threshold value

for readout of constructions being assembled in WM – a lower threshold value would result in fragmented phrases

whereas a higher value would yield more complete sentences. Also, preliminary evidence from an eye-tracking

experiment is discussed.


Vision: Links to Vanduffel and the challenge of finding neural correlates. Relation to Kemmerer.

Eye movements and EyeParser. Link to neuroinformatics?

Compare/contrast Embodied Construction Grammar (Schilling) and Fluid Construction Grammar

Session 3

Bornkessel-Schlesewsky: Language diversity, actor-driven comprehension and consequences

for the neural language architecture With over 6000 languages spoken in the world today, one of the most striking characteristics of human language is

surely the diversity of its manifestations. In view of this high variability, it is perhaps not surprising that absolute

linguistic universals are virtually non-existent. However, linguistic variation is also not arbitrary, with some patterns

occurring more frequently than others. These skewings towards certain patterns are likely tied to the way in which

language is implemented in the brain.

Building on this working hypothesis, I will present salient aspects of the latest version of a neurolinguistic

model of language processing, the extended Argument Dependency Model (eADM). The eADM follows in the

footsteps of MacWhinney and Bates' Competition Model in assuming that fine-grained linguistic cues such as

animacy, case marking, word order etc. mediate sentence comprehension across languages. It goes beyond the

Competition Model in positing that the use of these cues follows from an underlying functional principle: to find the

25

actor (the participant primarily responsible for the state of affairs) of a given proposition as quickly and

unambiguously as possible. This processing strategy likely follows from independent principles of goal-directed

action and possibly builds on the self vs. other distinction, with the self as the actor prototype. It is also compatible

with a range of typological generalisations including patterns of language change.

Competition for actorhood accounts for a range of phenomena that purely quantitative cue variation cannot

derive: changes in cue strength over the course of a sentence; qualitatively different brain responses to similar

phenomena in different languages; qualitatively similar brain responses to cues of vastly differing strengths in

different languages. These observations are based on a range of ERP studies conducted in typologically diverse

languages.

While previous versions of the eADM essentially provided a descriptive framework which could derive

these cross-linguistic results, the eADM 4.0 (Bornkessel-Schlesewsky & Schlesewsky, in preparation) spells out the

consequences of actor-oriented comprehension for the processing architecture as a whole, from word category

recognition to integration into the broader discourse. Salient features include:

� no lexical specification of word categories

� representation of word and sentence meaning via category neutral actor/action schemas

� tight integration between sentence and discourse level (reference specification), while maintaining

qualitative distinctions between the two

� clear specification of the relation between actor and "subject" (privileged syntactic argument); subject as a

cataphoric discourse relation

I will show how these architectural assumptions (a) derive existing ERP data; (b) suggest very general (and unified)

interpretations of language-related ERP components such as the N400 and P600; (c) make predictions for

neuroanatomy.


Compare/contrast Kempen’s unification model and MacWhinney’s competition model

Relation between actor/action schemas and SemRep?

Is actorhood really enough to serve as a backbone of the whole range of linguistic comprehension? Cf.

Kemmerer.

How do the ERP components/language processing steps relate to other non-linguistic cognitive capacities?

How can ERP data best be integrated with fMRI data?

Barres: Viewing Template Construction Grammar as a Model of Comprehension The talk will offer a comprehension view of TCG in relation to eADM; giving a dual path model in relation to

aphasia involving the deep semantics of world knowledge versus the semantic categories in constructions in building

a SemRep from an utterance.


Relation to Role and Reference grammar.

Relation to FCG

Compare/contrast the tight coupling of X-schemas into ECG

Developing a Workspace to support this kind of modeling.

26

Grosvald: Dissociating linguistic and non-linguistic gesture processing: An ERP study of

American Sign Language A fundamental advance in our understanding of human language would come from a detailed account of how

non-linguistic and linguistic information is differentiated in real time by language users. To explore this issue, we

targeted the N400, an ERP component known to be sensitive to semantic context. Deaf signers saw 120 American

Sign Language sentences, each consisting of a sentence frame (e.g. BOY SLEEP IN HIS) completed by one of four

items: a semantically congruent sign (e.g. BED), a semantically incongruent sign (e.g. LEMON), a pseudo-sign

(phonologically legal but non-lexical form), or a non-linguistic grooming gesture (e.g. the performer scratching her

face). To the extent that semantic processing is similar for spoken and signed language, a larger N400 response is

expected in the semantically congruent and pseudo-sign contexts than for the semantically congruent sentences. The

response to the non-linguistic grooming gestures is of particular interest: if processed linguistically during early

processing stages, such items might also be expected to elicit an N400-like component. On the other hand, a

different response should be expected if such items are rejected as non-linguistic at early stages of processing.

We found significant N400-like responses in the incongruent and pseudo-sign contexts while in contrast, the

gestures elicited a large positive-going waveform. These findings suggest that deaf signers can quickly detect and

disregard non-linguistic manual actions during real time language processing. Moreover, they represent an important

step in understanding the relationship between the processing of language and of human actions in general, and offer

information about the time-course and neural topography of these processes.


Relation to use of ERPs in the study of spoken language.

What is specific to language versus generic to action perception?

Relation to Cartmill’s presentation.

Challenges for comparing speech and sign in seeking neural correlates.

Cartmill: From ape and human gesture to language Gesture use in human and non-human primates; imitation and ritualization of signs; sensitivity to social contexts.

The relation between gesture and speech in human infants. The role of human gesture in grounding mental

representation in action. Additional Discussion Points for Wednesday

Differences between wild and human-reared apes

Apes' responding to and manipulating others' attentional states

Semantics of gestures and their use as symbols

Relation to Lee’s SemRep

Challenge of creating databases for gestural data and configuring them as targets for modeling?

Schilling: Recruitment of Grounded Internal Models in Action, Cognition and Language Following the behavior-based approach, we believe that behavior, which is usually and sensibly being described

on a global level, emerges from the cooperation of small neuronal behavioral elements characterized at a lower level

of description without requiring a central superior controller. We started this approach by concentrating on a fairly

27

complex behavior, hexapod walking, and are now continuing this approach by treating phenomena usually attributed

to cognitive systems.

Based on a behaviour-based system we are now developing a system showing cognitive abilities while still

strongly relying on reactive elements already given. To be able to expand the Walknet for gaining properties

attributed to cognitive systems, we require the introduction of further elements. Essential is the introduction of a

manipulable body model that allows the system to test the outcome of a given behavior via internal simulation, i.e.

without explicitly performing a, possibly risky, behavior. The next step concerns a small network that allows the

system to “concentrate” on specific, randomly selected behavioral elements, the effect of which can then be tested

via internal simulation. In this way, the system is creative and is able to plan ahead, two essential properties for a

system called “cognitive”.

Finally, this model shall be extended towards language. Language recruits the underlying system and language

understanding utilizes the internal simulation. The different levels of representation appear to be tightly connected.

Therefore, on the one hand, the structure of actions shows up in how language is structured. And, on the other hand,

language information can directly be integrated into a grounded conceptual system. In our approach, we want to

connect these different levels of representations (actions given as dynamic networks and language described through

constructions) and apply them in a communicative scenario.


How does language reflect the structure of action?

Connection to Kemmerer: Structure of Actions - structure of verbs

How are different types of knowledge integrated (e.g., information from language and conceptual knowledge)?

How can action controller architectures be adapted for embodied construction grammar?

Links between ECG and TCG and FCG?

Possible links to Kemmerer

MacWhinney: The Competition Model of language processing Progress on the Unified Model is still a work in progress. The top level description is intended to bring together

existing theories into neurolinguistic contact. Modeling work is focused on the lexicon (DevLex) and syntax (item-

based patterns); non-modeling work addresses the theory of perspective. The approach views Broca's area as gating

the ability of the lexicon to trigger motor commands.


Relation to Kemmerer’s paper?

Address Bornkessel’s 2006 comments on the competition model: On-line versus off-line; one language at a time

versus bilingual.

What does leaning of bilinguals suggest for the overall process of language acquisition and comprehension?

What are the neurolinguistic evidence and implications of the suggested models, especially in relation to other

neurolinguistics researchers of the Workshop?

The CHILDES database for language acquisition as a paradigm for behavioral databases. Issue: How are data to

be summarized into a form suitable for modeling?

28

Burns: Building a Breakthrough Machine Scientific breakthroughs are both elusive and massively significant to humanity. Given that the knowledge

architecture of scientific work is itself evolving based on developments in informatics, computing, text mining and

automated acquisition methods, can we re-examine how this architecture could be designed to understand what

constitutes a scientific breakthrough? Could we even design this architecture to generate breakthroughs? I will

describe a preliminary initiative challenge program at USC's Information Sciences Institute and invite attendees to

contribute and participate. This will involve worked examples from neuroscience based on neural connectivity data

that makes use of methods for extracting information from the full-text of neuroscience publications and reasoning

over that knowledge in an online system.

Session 4

Bohland: Informatics efforts to link gene expression data to speech/language brain systems The architecture of language-related brain systems can be described at

multiple scales, from genes and gene products that guide brain development and provide the molecular signatures

of populations of neurons to the circuits that comprise functional subsystems. While much of our neurobiologically

based experimental data related to language comes from neuroimaging or lesion studies, new efforts are now

enabling connections to be made between genes and gene sets and the higher-level organization of these brain

systems. I will describe such efforts in mouse, and now human, and discuss a new related web database aimed at

neurolinguistics researchers that we are developing. I will also discuss the issue of brain atlas concordance, which is

important in unifying diverse human brain datasets.


How does this new resource complement existing resources (classical atlases, connectivity data, activation foci

data) and what are the potential integration points between them?

How can the computational modeler make use of these new resources?

What would go in a Gene Expression SED?

Relation to Marques on the linkage of Brain Navigator to the Allen Brain Atlas.

Relation to Bota on comparison of human and macaque data.

Miikkulainen: Language and neural networks The GLIDES model, a neural network architecture that shows how the symbol-grounding problem can be solved through

learned relationships between simple visual scenes and linguistic descriptions. DISCERN, a connectionist model of human story

processing, as a framework for placing sentences in a larger context by putting together different memory systems.


Relating the GLIDES model to Lee’s work.

Using DISCERN to frame broader discussion of the role of context in analyzing sentences.

What kind of neural network (subsymbolic) modeling techniques can be used for implementing models of

Construction Grammar (e.g. TCG, FCG) and other approaches (e.g. the Unification model)?

How can those applied neural network components be linked to neuro-physiological circuitry of the brain?

29

Oztop: From the concept of ‘self-observation for mirror neuron development’ to imitation and

robot skill transfer The talk will introduce the self-observation mechanism for visuomotor development in the context of mirror

neuron formation and imitation learning. Self-observation allows a system to develop internal models than can help

predict future and eliminate delays in sensory processing. There is a range of evidence showing that our brain uses

this strategy elegantly for dealing with novel tools, agency detection and body image formation. As a utilization of

this capacity of the human brain, recently we proposed a skill transfer paradigm for artificial agents. Primates, in

particular humans, are very adept at learning to use tools. The paradigm uses this sensorimotor learning capacity to

obtain agent behaviors, which otherwise would require manual programming by experts. The idea is to consider the

target agent as a tool that can be controlled by a human. Provided with an intuitive interface for controlling the agent

(e.g. a robot), the human learns to perform a given task using the agent. This is akin to the stage where a beginner is

learning to drive a car. The key point in the proposed paradigm is that the human is included in the real-time control

loop, therefore his sensorimotor learning mechanisms is fully engaged allowing the human brain to act as an

adaptive controller to accomplish a given task. Once the control proficiency has been attained by the human, then

obtaining an autonomous controller boils down to reverse engineering the control policy established by the human

brain. After introducing and exemplifying the paradigm, I will present some ideas on the brain areas that might

mediate the capacity of humans to subsume an external agent into the body schema. Finally, the fact that these areas

can be considered as parts of the mirror system will be elaborated and linked back to the mirror neurons and their

possible function in the human and non-human primates.


Relation to Bonaiuto. Affordances.

Relation to Demiris.

Issues of the databases required to support such modeling.

Linking Agency (in neuroscience) with the Subject (linguistics) of a sentence

What do we need to make an artificial system (i.e. robot) build up a (proto)language?

How to bridge the gap between sensorimotor experiences of an agent with the symbols of the language?

Gasser: Ontogenetic Ritualization in nonhuman primates Ontogenetic ritualization is a process whereby an action between two conspecifics becomes replaced by a

communicative symbol derived from aspects of the action episode:

• Individual A performs behavior X.

• Individual B reacts consistently with behavior Y.

• Subsequently B anticipates A's performance of the complete behavior X by performing Y after observing only

some initial segment X’ of X.

• Subsequently, A anticipates B's anticipation and produces the initial step in a ritualized form XR (waiting for a

response) in order to elicit Y.

It has been proposed to explain the development of specific gestures in great ape communities. Here, we will

describe several archetypal examples of ontogenetic ritualization. We then outline a computational model that

explains the steps to go from episodes of direct interaction to physically separated communicative acts directed

30

towards similar goals. In so doing, we will briefly review a set of models capturing primate action recognition and

behavioral control systems and suggest the relevant extensions necessary to capture the degree of learning and

flexible behavioral control that ontogenetic ritualization requires.


Developing a Workspace to support this kind of modeling.

Wood: How cognition functions in primates Humans make inferences about other individuals’ intentions and goals by evaluating their actions in relation to

the constraints imposed by the environment. This capacity enables humans to go beyond the surface appearance of

behavior to draw inferences about an individual’s mental states. In this talk, I describe experiments showing that this

ability is shared with nonhuman primates. I then describe a new project designed to characterize the development of

action comprehension by combining controlled-rearing experiments of animals with computational modeling based

on Bayesian inverse planning.


Comparison of humans and primates.

Event representation in relation to SemRep.

Limitations of simulation theory

Inferential reasoning

What developmental studies can add to such research

Fox: BrainMap, ALE and meta-analytic modeling.��Current status of the BrainMap database, ALE (Activation Likelihood Estimation) and related tools for

coordinate-based meta-analysis of neural systems. These tools allow construction of graphical models of neural

systems including regions (nodes), connections (edges), and functional characterizations. Meta-analytic models have

high intrinsic information content and also increase statistical power when used as starting models for causal and

graphical modeling techniques.

Additional Discussion Points for Wednesday:

As these meta-analytic neural models are computed from post hoc activation co-occurrence probabilities, how

closely do they reflect neural systems as defined by other methods?

Can these meta-analysis and modeling strategies be extrapolated to types of brain data other than activation foci?

What methods of data harvesting has BrainMap tried? What are the implications for adding SEDs to BODB.

Have the BrainMap methods been applied to neurolinguistics?

Can the BrainMap methods be related to comparable data from non-human primates on action, gesture, etc.?

Implications for adding a computational meta-analysis tool to BODB.�

Implications for adding SEDs to BODB.

Nielsen: The Brede database and new tools for meta-analysis This provides an introduction to the Brede tools for handling and analyzing the data that is associated a

neuroimaging study and how it interfaces with and can be included in other databases.


31

BredeWiki versus Database. How do you deal with data quality issues in a wiki – not just the quality of data in

the system but the quality of data return to the user during search/browse/navigation of your site.

How to deal with the issues of data ownership and sharing [within or beyond brain imagers]

Link to Burns.

Relate to Marques on the use of BrainLink in Brain Navigator.

Background for the Workshop: Abstracts of Selected Papers by Participants

Arbib

Arbib, M. A., N. Tinroongroj, Plangprasopchok, A. and Bonaiuto, J.J. (2011). Brain Operating Principles

and the Brain Operation Database (BODB): A Framework for Cumulative Modeling in Systems and

Cognitive Neuroscience.

Empirical neuroscience and computational neuroscience complement each other. The former benefits from

predictions and suggestions from modeling, while model validation rests on empirical data. Unfortunately, empirical

data and models lack an integrated neuroinformatics environment, yielding differences in data representation that

impede the computational integration of diverse data. There are no adequate tools for searching experiments and

protocols relevant to testing a given set of models, or for seeking models that can yield predictions relevant to a

certain body of experiments. We here introduce the Brain Operation Database, BODB, which is designed to provide

an environment in which modelers and experimentalists can work together by making use of their shared empirical

data, models and know-how. Our work in BODB uses the structural ontology of nested brain regions but our

contribution to neuroinformatics ontology is to introduce the complementary functional ontology of Brain Operating

Principles (BOPs). These facilitate federation by setting forth functional principles that can structure both models

and observed neural function. BODB is structured around three information entities – summaries of empirical data,

models and summaries of simulation results, and Brain Operating Principles (BOPs) – along with structures that

enhance entry searching and model comparison

Arbib, M. A. (2010). Mirror System Activity for Action and Language is Embedded in the Integration of

Dorsal & Ventral Pathways. Brain and Language 112(1): 12-24.

We develop the view that the involvement of mirror neurons in embodied experience grounds brain structures

that underlie language, but that many other brain regions are involved. We stress the cooperation between the dorsal

and ventral streams in praxis and language. Both have perceptual and motor schemas but the perceptual schemas in

the dorsal path are affordances linked to specific motor schemas for detailed motor control, whereas the ventral path

supports planning and decision making. This frames the hypothesis that the mirror system for words evolved from

the mirror system for actions to support words-as-phonological-actions, with semantics provided by the linkage to

neural systems supporting perceptual and motor schemas. We stress the importance of computational models which

can be linked to the parametric analysis of data and conceptual analysis of these models to support new patterns of

understanding of the data. In the domain of praxis, we assess the FARS model of the canonical system for grasping,

the MNS models for the mirror system for grasping, and the Augmented Competitive Queuing model that extends

the control of action to the opportunistic scheduling of action sequences and also offers a new hypothesis on the role

of mirror neurons in self action. Turning to language, we use Construction Grammar as our linguistic framework to

32

get beyond single words to phrases and sentences, and initiate analysis of what brain functions must complement

mirror systems to support this functionality.

Arbib, M. A. (2011). From Mirror Neurons to Complex Imitation in the Evolution of Language and Tool

Use. Annual Review of Anthropology 40.

The mirror system hypothesis suggests that evolution expanded a basic mirror system for grasping, in concert

with other brain regions first to support simple imitation (shared with the common ancestor of humans and great

apes) and thence to complex imitation (unique to the hominin line), which includes overimitation, the apparent

drawbacks of which are in fact essential to human skill transmission. These advances in praxis supported the

emergence of pantomime and thence protosign and protospeech. This capacity, we claim, was adequate for cultural

evolution to then yield language. We argue that Oldowan tool making corresponds to simple imitation and ape

gestural communication and Acheulean tool making corresponds to complex imitation and protolanguage, whereas

the explosion of innovations in tool making and social organization of the past 100,000 years correlates with the

emergence of language. Care is taken, however, to distinguish brain mechanisms for praxis from those supporting

language.

Arbib, M. A., J. Bonaiuto, et al. (2009). Tool use and the distalization of the end-effector. Psychol Res

73(4): 441-62.

We review recent neurophysiological data from macaques and humans suggesting that the use of tools extends

the internal representation of the actor’s hand, and relate it to our modeling of the visual control of grasping. We

introduce the idea that, in addition to extending the body schema to incorporate the tool, tool use involves

distalization of the end-effector from hand to tool. Different tools extend the body schema in different ways, with a

displaced visual target and a novel, task-specific processing of haptic feedback to the hand. This distalization is

critical in order to exploit the unique functional capacities engendered by complex tools.

Aziz-Zadeh

Aziz-Zadeh L, Sheng T, Gheytanchi A (2010) Common Premotor Regions for the Perception and

Production of Prosody and Correlations with Empathy and Prosodic Ability. PLoS ONE 5(1): e8759.

doi:10.1371/journal.pone.0008759

Prosody, the melody and intonation of speech, involves the rhythm, rate, pitch and voice quality to relay

linguistic and emotional information from one individual to another. A significant component of human social

communication depends upon interpreting and responding to another person’s prosodic tone as well as one’s own

ability to produce prosodic speech. However there has been little work on whether the perception and production of

prosody share common neural processes, and if so, how these might correlate with individual differences in social

ability. The production of prosody is well known to be a specialization of the premotor cortex, in particular the

inferior frontal gyrus (IFG), with emotional prosody more strongly activating the right hemisphere and linguistic

prosody more strongly activating the left hemisphere [1,2]. Research on the perception of prosody has largely

focused on the right temporal lobe. However, despite this emphasis, there is some indication that the premotor

cortex may also be involved [1,3,4]. Nevertheless, premotor contributions to prosody perception have not been well

studied. There is limited evidence that there may be common frontal areas active for both the perception and

production of prosody; patients with lesions to frontal areas seem to have difficulty with both the perception and

33

production of prosody [2]. However, these lesions are often very large and it is difficult to discern if the same brain

areas are utilized in the two tasks. If the same areas were to be involved, it may indicate that, at least under some

circumstances, the acoustic signals from another person’s prosodic speech are transformed into articulatory signals

in order to understand prosodic meaning. That is, it may imply that in order to understand someone else’s prosodic

intonation, we may utilize our own motor representations of how we would produce the given intonation.

Aziz-Zadeh, Wilson, Rizzolatti & Iacoboni (2006): Congruent Embodied Representations for Visually

Presented Actions and Linguistic Phrases Describing Actions, Current Biology 16, 1–6, September 19, 2006.

The thesis of embodied semantics holds that conceptual representations accessed during linguistic processing are,

in part, equivalent to the sensory-motor representations required for the enactment of the concepts. Here, using

fMRI, we tested the hypothesis that areas inhuman premotor cortex that respond both to the execution and

observation of actions—mirror neuron areas – are key neural structures in these processes. Participants observed

actions and read phrases relating to foot, hand, or mouth actions. In the premotor cortex of the left hemisphere, a

clear congruence was found between effector-specific activations of visually presented actions and of actions

described by literal phrases. These results suggest a key role of mirror neuron areas in the re-enactment of sensory-

motor representations during conceptual processing of actions invoked by linguistic stimuli.

Aziz-Zadeh, Koski, Zaidel, Mazziotta & Iacoboni (2006): Lateralization of the Human Mirror Neuron

System

A cortical network consisting of the inferior frontal, rostral inferior parietal, and posterior superior temporal

cortices has been implicated in representing actions in the primate brain and is critical to imitation in humans. This

neural circuitry may be an evolutionary precursor of neural systems associated with language. However, language is

predominantly lateralized to the left hemisphere, whereas the degree of lateralization of the imitation circuitry in

humans is unclear. We conducted a functional magnetic resonance imaging study of imitation of finger movements

with lateralized stimuli and responses. During imitation, activity in the inferior frontal and rostral inferior parietal

cortex, although fairly bilateral, was stronger in the hemisphere ipsilateral to the visual stimulus and response hand.

This ipsilateral pattern is at variance with the typical contralateral activity of primary visual and motor areas.

Reliably increased signal in the right superior temporal sulcus (STS) was observed for both left-sided and right-

sided imitation tasks, although subthreshold activity was also observed in the left STS. Overall, the data indicate that

visual and motor components of the human mirror system are not left-lateralized. The left hemisphere superiority for

language, then, must be have been favored by other types of language precursors, perhaps auditory or multimodal

action representations.

Lisa Aziz-Zadeh, Antonio Damasio, Embodied semantics for actions: Findings from functional brain

imaging Journal of Physiology - Paris 102 (2008) 35–39

The theory of embodied semantics states that concepts are represented in the brain within the same sensory-

motor circuitry in which the enactment of that concept relies. For example, the concept of ‘‘grasping” would be

represented in sensory-motor areas that represent grasping actions; the concept of ‘‘kicking” would be represented

by sensory-motor areas that control kicking actions; and so forth. This theory indicates that the secondary sensory-

motor areas, which are known to be involved in mental simulation of world experiences (Rizzolatti and Craighero,

2004), may be responsible for the representation of concepts (Gallese and Lakoff, 2005). Instead, primary sensory-

34

motor areas are presumed to be inhibited, in order to distinguish between a general representation of a given concept

and its actualization. This theory has been extended by Lakoff and his colleagues to include metaphors. Thus the

phrase ‘‘kick off the year” would also involve the motor representations related to kicking, just as the phrase ‘‘grasp

the explanation” would involve motor representations related to control of the hand (Lakoff and Johnson, 1999).

Ideas along similar lines have been discussed previously by other groups (for example, Barsalou, 1999; Damasio,

1989; Damasio and Tranel, 1993; Feldman and Narayanan, 2004; Glenberg and Kaschak, 2002; Pulvermuller, 2005;

Pulvermuller et al., 2005). While embodied semantics would apply to all concepts, (i.e., concepts associated with

visual movement would be represented by visual motion areas such as MT, etc.), most efforts to date have been

placed on investigating the neural correlates of concepts associated with actions (kicking, grasping, biting, etc.).

Here we review first some neuropsychological literature investigating the link between language and motor

processing, then move to the link between embodied semantics for language and the mirror neuron system and fMRI

data on embodied semantics for actions. Next we consider representations for language and for actions in BA 44,

and finally embodied semantics for metaphors.

Aziz-Zadeh, Lisa, Fiebach, Christian J., Naranayan, Srini, Feldman, Jerome, Dodge, Ellen and Ivry,

Richard B. (2007) 'Modulation of the FFA and PPA by language related to faces and places', Social

Neuroscience, 1 – 10

Does sentence comprehension related to faces modulate activity in the fusiform face area (FFA) and does

sentence comprehension related to places modulate activity in the parahippocampal place area (PPA)? We

investigated this question in an fMRI experiment. Participants listened to sentences describing faces, places, or

objects, with the latter serving as a control condition. In a separate run, we localized the FFA and PPA in each

participant using a perceptual task. We observed a significant interaction between the region of interest (FFA vs.

PPA) and sentence type (face vs. place). Activity in the left FFA was modulated by face sentences and in the left

PPA was modulated by place sentences. Surprisingly, activation in each region of interest was reduced when

listening to sentences requiring semantic analysis related to that region’s domain specificity. No modulation was

found in the corresponding right hemisphere ROIs. We conclude that processing sentences may involve inhibition of

some visual processing areas in a content-specific manner. Furthermore, our data indicate that this semantic-based

modulation is restricted to the left hemisphere. We discuss how these results may constrain neural models of

embodied semantics.

Bohland

The Brain Atlas Concordance Problem: Quantitative Comparison of Anatomical Parcellations

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0007200

Many neuroscientific reports reference discrete macro-anatomical regions of the brain which were delineated

according to a brain atlas or parcellation protocol. Currently, however, no widely accepted standards exist for

partitioning the cortex and subcortical structures, or for assigning labels to the resulting regions, and many

procedures are being actively used. Previous attempts to reconcile neuroanatomical nomenclatures have been largely

qualitative, focusing on the development of thesauri or simple semantic mappings between terms. Here we take a

fundamentally different approach, discounting the names of regions and instead comparing their definitions as

spatial entities in an effort to provide more precise quantitative mappings between anatomical entities as defined by

35

different atlases. We develop an analytical framework for studying this brain atlas concordance problem, and apply

these methods in a comparison of eight diverse labeling methods used by the neuroimaging community. These

analyses result in conditional probabilities that enable mapping between regions across atlases, which also form the

input to graph-based methods for extracting higher-order relationships between sets of regions and to procedures for

assessing the global similarity between different parcellations of the same brain. At a global scale, the overall results

demonstrate a considerable lack of concordance between available parcellation schemes, falling within chance levels

for some atlas pairs. At a finer level, this study reveals spatial relationships between sets of defined regions that are

not obviously apparent; these are of high potential interest to researchers faced with the challenge of comparing

results that were based on these different anatomical models, particularly when coordinate-based data are not

available. The complexity of the spatial overlap patterns revealed points to problems for attempts to reconcile

anatomical parcellations and nomenclatures using strictly qualitative and/or categorical methods. Detailed results

from this study are made available via an interactive web site at http://obart.info.

An anatomic gene expression atlas of the adult mouse brain

http://www.nature.com/neuro/journal/v12/n3/abs/nn.2281.html?lang=en

Studying gene expression provides a powerful means of understanding structure-function relationships in the

nervous system. The availability of genome-scale in situ hybridization datasets enables new possibilities for

understanding brain organization based on gene expression patterns. The Anatomic Gene Expression Atlas (AGEA)

is a new relational atlas revealing the genetic architecture of the adult C57Bl/6J mouse brain based on spatial

correlations across expression data for thousands of genes in the Allen Brain Atlas (ABA). The AGEA includes

three discovery tools for examining neuroanatomical relationships and boundaries: (1) three-dimensional

expression-based correlation maps, (2) a hierarchical transcriptome-based parcellation of the brain and (3) a facility

to retrieve from the ABA specific genes showing enriched expression in local correlated domains. The utility of this

atlas is illustrated by analysis of genetic organization in the thalamus, striatum and cerebral cortex. The AGEA is a

publicly accessible online computational tool integrated with the ABA (http://mouse.brain-map.org/agea).

Clustering of spatial gene expression patterns in the mouse brain and comparison with classical

neuroanatomy

http://www.sciencedirect.com/science/article/pii/S1046202309002035

Spatial gene expression profiles provide a novel means of exploring the structural organization of the brain.

Computational analysis of these patterns is made possible by genome-scale mapping of the C57BL/6J mouse brain

in the Allen Brain Atlas. Here we describe methodology used to explore the spatial structure of gene expression

patterns across a set of 3041 genes chosen on the basis of consistency across experimental observations (N = 2). The

analysis was performed on smoothed, co-registered 3D expression volumes for each gene obtained by aggregating

cellular resolution image data. Following dimensionality and noise reduction, voxels were clustered according to

similarity of expression across the gene set. We illustrate the resulting parcellations of the mouse brain for different

numbers of clusters (K) and quantitatively compare these parcellations with a classically-defined anatomical

reference atlas at different levels of granularity, revealing a high degree of correspondence. These observations

suggest that spatial localization of gene expression offers substantial promise in connecting knowledge at the

molecular level with higher-level information about brain organization.

36

Neural Representations and Mechanisms for the Performance of Simple Speech Sequences Jason W.

Bohland, Daniel Bullock, and Frank H. Guenther, Journal of Cognitive Neuroscience 22:7, pp. 1504–1529

Here we present a neural model that describes how the brain may represent and produce sequences of simple,

learned speech sounds. This model addresses the question of how, using a finite inventory of learned speech motor

actions, a speaker can produce arbitrarily many utterances that fall within the phonotactic and linguistic rules of her

language. At the phonological level of representation, the model implements two complementary subsystems,

corresponding to the structure and content of planned speech utterances within a neurobiologically realistic

framework that simulates interacting cortical and subcortical structures. This phonological representation is

hypothesized to interface between the higher level conceptual and morphosyntactic language centers and the lower

level speech motor control system, which itself implements only a limited set of learned motor programs. In the

current formulation, syllable-sized representations are ultimately selected through phonological encoding, and these

activate the most appropriate sensorimotor programs, commanding the execution of the planned sound. Construction

of the model was guided by previous theoretical work as well as clinical and experimental results, most notably a

companion fMRI study (Bohland & Guenther, 2006).

Bonaiuto

Bonaiuto, J. B., E. Rosta, et al. (2007). Extending the mirror neuron system model, I : Audible actions and

invisible grasps. Biol Cybern 96: 9-38.

The paper introduces mirror neuron system II (MNS2), a new version of the MNS model (Oztop and Arbib in

Biol Cybern 87 (2):116-140, 2002) of action recognition learning by mirror neurons of the macaque brain. The new

model uses a recurrent architecture that is biologically more plausible than that of the original model. Moreover,

MNS2 extends the capacity of the model to address data on audio-visual mirror neurons and on the response of

mirror neurons when the target object was recently visible but is currently hidden.

Bonaiuto, J. B. and M. A. Arbib (2010). Extending the mirror neuron system model, II: What did I just

do? A new role for mirror neurons. Biological Cybernetics 102: 341-359.

A mirror system is active both when an animal executes a class of actions (self-actions) and when it sees another

execute an action of that class. Much attention has been given to the possible roles of mirror systems in responding

to the actions of others but there has been little attention paid to their role in self-actions. In the companion article

(Bonaiuto et al., 2007) we presented MNS2, an extension of the Mirror Neuron System (MNS) model of the monkey

mirror system trained to recognize the external appearance of its own actions as a basis for recognizing the actions

of other animals when they perform similar actions. Here we further extend the study of the mirror system by

introducing the novel hypotheses that a mirror system may additionally help in monitoring the success of a self-

action and may also be activated by recognition of one’s own apparent actions as well as efference copy from one’s

intended actions. The framework for this computational demonstration is a model of action sequencing, called

augmented competitive queuing, in which action choice is based on the desirability of executable actions. We show

how this what did I just do? function of mirror neurons can contribute to the learning of both executability and

desirability which in certain cases supports rapid reorganization of motor programs in the face of disruptions.

37

Bornkessel-Schlesewsky

Borkessel & Schlesewsky (2006) - The extended argument dependency model: a neurocognitive approach

to sentence comprehension across languages

Real-time language comprehension is a principal cognitive ability and thereby relates to central properties of the

human cognitive architecture. Yet how do the presumably universal cognitive and neural substrates of language

processing relate to the astounding diversity of human languages (over 5,000)? The authors present a neurocognitive

model of online comprehension, the extended argument dependency model (eADM), that accounts for cross-

linguistic unity and diversity in the processing of core constituents (verbs and arguments). The eADM postulates

that core constituent processing proceeds in three hierarchically organized phases: (1) constituent structure building

without relational interpretation, (2) argument role assignment via a restricted set of cross-linguistically motivated

information types (e.g., case, animacy), and (3) completion of argument interpretation using information from

further domains (e.g., discourse context, plausibility). This basic architecture is assumed to be universal, with cross-

linguistic variation deriving primarily from the information types applied in Phase 2 of comprehension. This

conception can derive the appearance of similar neurophysiological and neuroanatomical processing correlates in

seemingly disparate structures in different languages and, conversely, of cross-linguistic differences in the

processing of similar sentence structures.

Bornkessel-Schlesewsky & Schlesewsky (2009) - Minimality as vacuous distinctness: Evidence from cross-

linguistic sentence comprehension

Psycholinguistic theorising has long been shaped by the assumption that the processing system endeavours to

minimise structures/relations during online comprehension. Within the scope of a recent cross-linguistic,

neurocognitive model of sentence comprehension (Bornkessel and Schlesewsky, 2006), we also proposed that the

assumption of a very general ‘Minimality’ principle can account for a variety of psycholinguistic findings from a

range of languages. In the present paper, we review empirical evidence for this notion of Minimality, before going

on to discuss its limitations. On the basis of this discussion, we propose that, rather than constituting an independent

processing principle, Minimality should be considered a subcase of a more general requirement for sentential

constituents to be distinct from one another. We show that this notion of Minimality as Distinctness (MaD) can

straightforwardly derive a wide range of findings on cross-linguistic sentence comprehension, while additionally

serving to simplify the overall processing architecture.

Bornkessel-Schlesewsky & Schlesewsky (2009) – The role of prominence information in the real-time

comprehension of transitive constructions: A cross-linguistic approach

Approaches to language processing have traditionally been formulated with reference to general cognitive

concepts (e.g. working memory limitations) of have based their representational assumptions on concepts from

linguistic theory (e.g. structure determines interpretation). Thus, many well-established generalizations about

language that have emerged from cross-linguistic/typological research have not as yet had a major influence in

shaping ideas about online processing. Here, we examine the viability of using typologically motivated concepts to

account for phenomena in online language comprehension. In particular, we focus on the comprehension of simple

transitive sentences (i.e. sentences involving two arguments/event participants) and cross-linguistics similarities and

differences in how they are processed. We argue that incremental argument interpretation in these structures is best

38

explained with reference to a range of cross-linguistically motivated, hierarchically ordered information types

termed ‘prominence scales’ (e.g. animacy, definiteness/specificity, case marking and linear order). We show that the

assumption of prominence-based argument processing can capture a wide range of recent neurocognitive findings,

as well as deriving well-known behavioral results.

Bornkessel-Schlesewsky et al. (2011) - Think globally: Cross-linguistic variation in electrophysiological

activity during sentence comprehension

This paper demonstrates systematic cross-linguistic differences in the electrophysiological correlates of conflicts

between form and meaning (semantic reversal anomalies). These engender P600 effects in English and Dutch (e.g.

Kolk et al., 2003; Kuperberg et al., 2003), but a biphasic N400 - late positivity pattern in German (Schlesewsky and

Bornkessel-Schlesewsky, 2009), and monophasic N400 effects in Turkish (Experiment 1) and Mandarin Chinese

(Experiment 2). Experiment 3 revealed that, in Icelandic, semantic reversal anomalies show the English pattern with

verbs requiring a position-based identification of argument roles, but the German pattern with verbs requiring a

case-based identification of argument roles. The overall pattern of results reveals two separate dimensions of cross-

linguistic variation: (i) the presence vs. absence of an N400, which we attribute to cross-linguistic differences with

regard to the sequence-dependence of the form-to-meaning mapping and (ii) the presence vs. absence of a late

positivity, which we interpret as an instance of a categorisation-related late P300, and which is observable when the

language under consideration allows for a binary well-formedness categorisation of reversal anomalies. We

conclude that, rather than reflecting linguistic domains such as syntax and semantics, the late positivity vs. N400

distinction is better understood in terms of the strategies that serve to optimise the form-to-meaning mapping in a

given language.

Bornkessel-Schlesewsky et al. (2010) - Prominence vs. aboutness in sequencing: A functional distinction

within the left inferior frontal gyrus

Prior research on the neural bases of syntactic comprehension suggests that activation in the left inferior frontal

gyrus (lIFG) correlates with the processing of word order variations. However, there are inconsistencies with respect

to the specific subregion within the IFG that is implicated by these findings: the pars opercularis or the pars

triangularis. Here, we examined the hypothesis that the dissociation between pars opercularis and pars triangularis

activation may reflect functional differences between clause-medial and clause-initial word order permutations,

respectively. To this end, we directly compared clause-medial and clause-initial object-before-subject orders in

German in a within-participants, event-related fMRI design. Our results showed increased activation for object-

initial sentences in a bilateral network of frontal, temporal and subcortical regions. Within the lIFG, posterior and

inferior subregions showed only a main effect of word order, whereas more anterior and superior subregions showed

effects of word order and sentence type, with higher activation for sentences with an argument in the clause-initial

position. These findings are interpreted as evidence for a functional gradation of sequence processing within the left

IFG: posterior subportions correlate with argument prominence-based (local) aspects of sequencing, while anterior

subportions correlate with aboutness-based aspects of sequencing, which are crucial in linking the current sentence

to the wider discourse. This proposal appears compatible with more general hypotheses about information

processing gradients in prefrontal cortex (Koechlin & Summerfield, 2007).

39

Bota

For the Talk:

Brain architecture management system

http://www.springerlink.com/content/w158346256g71u47/

Abstract: The nervous system can be viewed as a biological computer whose genetically determined

macrocircuitry has two basic classes of parts: gray matter regions interconnected by fiber pathways. We describe

here the basic features of an online knowledge management system for storing and inferring relationships between

data about the structural organization of nervous system circuitry. It is called the Brain architecture management

system (BAMS; http://brancusi.usc.edu/bkms) and it stores and analyzes data specifically concerned with

nomenclature and its hierarchical taxonomy, with axonal connections between regions, and with the neuronal cell

types that form regions and fiber pathways.

Bota M. & Arbib. M.A. (2004), Neuroinformatics 2(1): 19-58. Language evolution: neural homologies and

neuroinformatics

http://www.sciencedirect.com/science/article/pii/S0893608003002399

Abstract: This paper contributes to neurolinguistics by grounding an evolutionary account of the readiness of the

human brain for language in the search for homologies between different cortical areas in macaque and human. We

consider two hypotheses for this grounding, that of Aboitiz and García [Brain Res. Rev. 25 (1997) 381] and the

Mirror System Hypothesis of Rizzolatti and Arbib [Trends Neurosci. 21 (1998) 188] and note the promise of

computational modeling of neural circuitry of the macaque and its linkage to analysis of human brain imaging data.

In addition to the functional differences between the two hypotheses, problems arise because they are grounded in

different cortical maps of the macaque brain. In order to address these divergences, we have developed several

neuroinformatics tools included in an on-line knowledge management system, the NeuroHomology Database, which

is equipped with inference engines both to relate and translate information across equivalent cortical maps and to

evaluate degrees of homology for brain regions of interest in different species.

For future discussion, add:

From gene networks to brain networks

http://www.nature.com/neuro/journal/v6/n8/abs/nn1096.html

Abstract: The brain's structural organization is so complex that 2,500 years of analysis leaves pervasive

uncertainty about (i) the identity of its basic parts (regions with their neuronal cell types and pathways

interconnecting them), (ii) nomenclature, (iii) systematic classification of the parts with respect to topographic

relationships and functional systems and (iv) the reliability of the connectional data itself. Here we present a

prototype knowledge management system (http://brancusi.usc.edu/bkms/) for analyzing the architecture of brain

networks in a systematic, interactive and extendable way. It supports alternative interpretations and models, is based

on fully referenced and annotated data and can interact with genomic and functional knowledge management

systems through web services protocols.

Swanson L.W. & Bota M. (2010), Proc Natl Acad Sci U S A. Foundational model of structural connectivity in the nervous system with a schema for wiring diagrams, connectome, and basic plan architecture.

http://www.pnas.org/content/107/48/20610.long

40

The nervous system is a biological computer integrating the body's reflex and voluntary environmental

interactions (behavior) with a relatively constant internal state (homeostasis)—promoting survival of the individual

and species. The wiring diagram of the nervous system's structural connectivity provides an obligatory foundational

model for understanding functional localization at molecular, cellular, systems, and behavioral organization levels.

This paper provides a high-level, downwardly extendible, conceptual framework—like a compass and map—for

describing and exploring in neuroinformatics systems (such as our Brain Architecture Knowledge Management

System) the structural architecture of the nervous system's basic wiring diagram. For this, the Foundational Model of

Connectivity's universe of discourse is the structural architecture of nervous system connectivity in all animals at all

resolutions, and the model includes two key elements—a set of basic principles and an internally consistent set of

concepts (defined vocabulary of standard terms)—arranged in an explicitly defined schema (set of relationships

between concepts) allowing automatic inferences. In addition, rules and procedures for creating and modifying the

foundational model are considered. Controlled vocabularies with broad community support typically are managed

by standing committees of experts that create and refine boundary conditions, and a set of rules that are available on

the Web.

Burns

Knowledge management of the neuroscientific literature: the data model and underlying strategy of the

NeuroScholar system

http://rstb.royalsocietypublishing.org/content/356/1412/1187.short

This paper describes the underlying strategy and system's design of a knowledge management system for the

neuroscientific literature called ‘NeuroScholar’. The problem that the system is designed to address is to delineate

fully the neural circuitry involved in a specific behaviour. The use of this system provides experimental

neuroscientists with a new method of building computational models (‘knowledge models’) of the contents of the

published literature. These models may provide input for analysis (conceptual or computational), or be used as

constraint sets for conventional neural modelling work. The underlying problems inherent in this approach, the

general framework for the proposed solution, the practical issues concerning usage of the system and a detailed,

technical account of the system are described. The author uses a widely used software specification language (the

Universal Modelling Language) to describe the design of the system and present examples from published work

concerned with classical eyeblink conditioning in the rabbit.

Tools and approaches for the construction of knowledge models from the neuroscientific literature

http://www.springerlink.com/content/8t786482928v5q77/

Within this paper, we describe a neuroinformatics project (called NeuroScholar, http://www.neuroscholar.org/)

that enables researchers to examine, manage, manipulate, and use the information contained within the published

neuroscientific literature. The project is built within a multi-level, multi-component framework constructed with the

use of software engineering methods that themselves provide code-building functionality for neuroinformaticians.

We describe the different software layers of the system. First, we present a hypothetical usage scenario illustrating

how NeuroScholar permits users to address largescale questions in a way that would otherwise be impossible. We

do this by applying NeuroScholar to a real-world neuroscience question: How is stress-related information

processed in the brain? We then explain how the overall design of NeuroScholar enables the system to work and

41

illustrate different components of the user interface. We then describe the knowledge management strategy we use

to store interpretations. Finally, we describe the software engineering framework we have devised (called the View-

Primitive-Data Model framework, [VPDMf]) to provide an open-source, accelerated software development

environment for the project. We believe that NeuroScholar will be useful to experimental neuroscientists by helping

them interact with the primary neuroscientific literature in a meaningful way, and to neuroinformaticians by

providing them with useful, affordable software engineering tools.

Intelligent Approaches to Mining the Primary Research Literature: Techniques, Systems, and Examples

http://www.springerlink.com/content/u5h203h351q74887/

In this chapter, we describe how creating knowledge bases from the primary biomedical literature is formally

equivalent to the process of performing a literature review or a ‘research synthesis’. We describe a principled

approach to partitioning the research literature according to the different types of experiments performed by

researchers and how knowledge engineering approaches must be carefully employed to model knowledge from

different types of experiment. The main body of the chapter is concerned with the use of text mining approaches to

populate knowledge representations for different types of experiment. We provide a detailed example from

neuroscience (based on anatomical tract-tracing experiments) and provide a detailed description of the methodology

used to perform the text mining itself (based on the Conditional Random Fields model). Finally, we present data

from textmining experiments that illustrate the use of these methods in a real example. This chapter is designed to

act as an introduction to the field of biomedical text-mining for computer scientists who are unfamiliar with the way

that biomedical research uses the literature.

Cartmill

Cartmill & Byrne (2010): Semantics of primate gestures: intentional meanings of orangutan gestures

Great ape gesture has become a research topic of intense interest, because its intentionality and flexibility suggest

strong parallels to human communication. Yet the fundamental question of whether an animal species’ gestures

carry specific meanings has hardly been addressed. We set out a systematic approach to studying intentional

meaning in the gestural communication of non-humans and apply it to a sample of orangutan gestures. We propose

that analysis of meaning should be limited to gestures for which (1) there is strong evidence for intentional

production and (2) the recipient’s final reaction matches the presumed goal of the signaller, as determined

independently. This produces a set of successful instances of gesture use, which we describe as having goal–

outcome matches. In this study, 28 orangutans in three European zoos were observed for 9 months. We

distinguished 64 gestures on structural grounds, 40 of which had frequent goal–outcome matches and could

therefore be analysed for meaning. These 40 gestures were used predictably to achieve one of 6 social goals: to

initiate an affiliative interaction (contact, grooming, or play), request objects, share objects, instigate co-locomotion,

cause the partner to move back, or stop an action. Twenty-nine of these gestures were used consistently with a single

meaning. We tested our analysis of gesture meaning by examining what gesturers did when the response to their

gesture did not match the gesture’s meaning. Subsequent actions of the gesturer were consistent with our

assignments of meaning to gestures. We suggest that, despite their contextual flexibility, orangutan gestures are

made with the expectation of specific behavioural responses and thus have intentional meanings as well as

functional consequences.

42

Cartmill & Byrne (2007): Orangutans Modify Their Gestural Signaling According to Their Audience's

Comprehension

When people are not fully understood, they persist with attempts to communicate, elaborating their speech in

order to better convey their meaning [1]. We investigated whether captive orangutans (Pongo pygmaeus and Pongo

abelii) would use analogous communicative strategies in signaling to a humanexperimenter, and whether they could

distinguish different degrees of misunderstanding. Orangutans' behavior varied according to how well they had

apparently been understood. When their aims were not met, they persisted in communicative attempts. However,

when the interlocutor appeared partially to understand their meaning, orangutans narrowed down their range of

signals, focusing on gestures already used and repeating them frequently. In contrast, when completely

misunderstood, orangutans elaborated their range of gestures, avoiding repetition of failed signals. It is therefore

possible, from communicative signals alone, to determine how well an orangutan's intended goal has been met. This

differentiation might function under natural conditions to allow an orangutan's intended goals to be understood more

efficiently. In the absence of conventional labels, communicating the fact that an intention has been somewhat

misunderstood is an important way to establish shared meaning.

Cartmill, E., Beilock, S., & Goldin-Meadow, S. (in press) A word in the hand: Action, gesture, and mental

representation in humans and non-human primates. Philosophical Transactions of the Royal Society B

(Special Issue “Action to Language”)

The movements we make with our hands both reflect our mental processes and help to shape them. Our actions

and gestures can affect our mental representations of actions and objects. In this paper, we explore the relationship

between action, gesture, and thought in both humans and non-human primates and discuss its role in the evolution of

language. Human gesture (specifically representational gesture) may provide a unique link between action and

mental representation. It is kinesthetically close to action and is, at the same time, symbolic. Non-human primates

use gesture frequently to communicate, and do so flexibly. However, their gestures mainly resemble incomplete

actions and lack the representational elements that characterize much of human gesture. Differences in the mirror

neuron system provide a potential explanation for non-human primates’ lack of representational gestures; the

monkey mirror system does not respond to representational gestures, while the human system does. In humans,

gesture grounds mental representation in action, but there is no evidence for this link in other primates. We argue

that gesture played an important role in the transition to symbolic thought and language in human evolution,

following a cognitive leap that allowed gesture to incorporate representational elements.

Cartmill, E. A., Demir, Ö. E. & Goldin-Meadow, S. (in press). Studying Gesture. In E. Hoff (Ed.) Research

Methods in Child Language: A Practical Guide, First Edition. Blackwell Publishing Ltd.

To gain a full understanding of the steps children follow in acquiring language, researchers must pay attention to

their hands as well as their mouths – that is, to gesture. We first define our methodology for studying gesture. We

then describe different types of gestures and their typical uses, and the methods by which meaning can be attributed

to gesture. We stress the importance of characterizing the relationship between gesture and speech, and illustrate

how that relationship changes over time as children’s spoken language develops. Importantly, the methods for

coding and analyzing gesture in relation to speech also change over time, and we provide examples of these

changes. We end by discussing gesture’s role in language learning and later stages of cognitive development.

43

Demiris

Demiris & Khadhouri (2008): Content-based control of goal-directed attention during human action

perception

http://www.iis.ee.ic.ac.uk/yiannis/DemirisKhadhouri08-IS.pdf

During the perception of human actions by robotic assistants, the robotic assistant needs to direct its

computational and sensor resources to relevant parts of the human action. In previous work we have introduced

HAMMER (Hierarchical Attentive Multiple Models for Execution and Recognition) (Demiris and Khadhouri,

2006), a computational architecture that forms multiple hypotheses with respect to what the demonstrated task is,

and multiple predictions with respect to the forthcoming states of the human action. To confirm their predictions, the

hypotheses request information from an attentional mechanism, which allocates the robot’s resources as a function

of the saliency of the hypotheses. In this paper we augment the attention mechanism with a component that

considers the content of the hypotheses’ requests, with respect to the content’s reliability, utility and cost. This

content-based attention component further optimises the utilisation of the resources while remaining robust to noise.

Such computational mechanisms are important for the development of robotic devices that will rapidly respond to

human actions, either for imitation or collaboration purposes.

Demiris (2007): Prediction of intent in robotics and multi-agent systems

http://www.iis.ee.ic.ac.uk/yiannis/DemirisCP07.pdf

Moving beyond the stimulus contained in observable agent behaviour, i.e. understanding the underlying intent of

the observed agent is of immense interest in a variety of domains that involve collaborative and competitive

scenarios, for example assistive robotics, computer games, robot–human interaction, decision support and intelligent

tutoring. This review paper examines approaches for performing action recognition and prediction of intent from a

multi-disciplinary perspective, in both single robot and multi-agent scenarios, and analyses the underlying

challenges, focusing mainly on generative approaches

Demiris & Simmons (2006): Perceiving the unusual: Temporal properties of hierarchical motor

representations for action perception

http://www.iis.ee.ic.ac.uk/yiannis/DemirisSimmonsNN06.pdf

Recent computational approaches to action imitation have advocated the use of hierarchical representations in the

perception and imitation of demonstrated actions. Hierarchical representations present several advantages, with the

main one being their ability to process information at multiple levels of detail. However, the nature of the

hierarchies in these approaches has remained relatively unsophisticated, and their relation with biological evidence

has not been investigated in detail, in particular with respect to the timing of movements. Following recent

neuroscience work on the modulation of the premotor mirror neuron activity during the observation of unpredictable

grasping movements, we present here an implementation of our HAMMER architecture using the minimum

variance model for implementing reaching and grasping movements that have biologically plausible trajectories.

Subsequently, we evaluate the performance of our model in matching the temporal dynamics of the modulation of

cortical excitability during the passive observation of normal and unpredictable movements of human

demonstrators.

Johnson & Demiris (2005): Perceptual Perspective Taking and Action Recognition

44

http://www.iis.ee.ic.ac.uk/yiannis/JohnsonDemiris-ijars05.pdf

Robots that operate in social environments need to be able to recognise and understand the actions of other

robots, and humans, in order to facilitate learning through imitation and collaboration. The success of the simulation

theory approach to action recognition and imitation relies on the ability to take the perspective of other people, so as

to generate simulated actions from their point of view. In this paper, simulation of visual perception is used to

recreate the visual egocentric sensory space and egocentric behaviour space of an observed agent, and through this

increase the accuracy of action recognition. To demonstrate the approach, experiments are performed with a robot

attributing perceptions to and recognising the actions of a second robot similar mechanisms for retaining different

types of visual information.

Y. Demiris and A. Meltzoff (2008) “The Robot in the Crib: A developmental analysis of imitation skills in

infants and robots”, Infant and Child Development, 17:43-53.

http://www.iis.ee.ic.ac.uk/yiannis/DemirisMeltzoff08-ICD.pdf

Interesting systems, whether biological or artificial, develop. Starting from some initial conditions, they respond

to environmental changes, and continuously improve their capabilities. Developmental psychologists have dedicated

significant effort to studying the developmental progression of infant imitation skills, because imitation underlies the

infant’s ability to understand and learn from his or her social environment. In a converging intellectual endeavor,

roboticists have been equipping robots with the ability to observe and imitate human actions because such abilities

can lead to rapid teaching of robots to perform tasks. We provide here a comparative analysis between studies of

infants imitating and learning from human demonstrators, and computational experiments aimed at equipping a

robot with such abilities. We will compare the research across the following two dimensions: (a) initial conditions -

what is innate in infants, and what functionality is initially given to robots, and (b) developmental mechanisms -

how does the performance of infants improve over time, and what mechanisms are given to robots to achieve

equivalent behavior. Both developmental science and robotics are critically concerned with: (a) how their systems

can and do go ‘beyond the stimulus’ given during the demonstration, and (b) how the internal models used in this

process are acquired over the lifetime of the system.

Fox

Publication links and pdfs may be found at: http://www.brainmap.org/

Eickhoff, S.B., Laird, A.R., Grefkes, C., Wang, L.E., Zilles, K., Fox, P.T., 2009. Coordinate-Based

Activation Likelihood Estimation Meta-Analysis of Neuroimaging Data: A Random-Effects Approach Based

on Empirical Estimates of Spatial Uncertainty. Human Brain Mapping 30, 2907- 2926.

A widely used technique for coordinate-based meta-analyses of neuroimaging data is activation likelihood

estimation (ALE). ALE assesses the overlap between foci based on modeling them as probability distributions

centered at the respective coordinates. In this Human Brain Project/Neuroinformatics research, the authors present a

revised ALE algorithm addressing drawbacks associated with former implementations. The first change pertains to

the size of the probability distributions, which had to be specified by the used. To provide a more principled

solution, the authors analyzed fMRI data of 21 subjects, each normalized into MNI space using nine different

approaches. This analysis provided quantitative estimates of between-subject and between-template variability for

16 functionally defined regions, which were then used to explicitly model the spatial uncertainty associated with

45

each reported coordinate. Secondly, instead of testing for an above-chance clustering between foci, the revised

algorithm assesses above-chance clustering between experiments. The spatial relationship between foci in a given

experiment is now assumed to be fixed and ALE results are assessed against a null-distribution of random spatial

association between experiments. Critically, this modification entails a change from fixed- to random-effects

inference in ALE analysis allowing generatization of the results to the entire population of studies analyzed. By

comparative analysis of real and simulated data, the authors showed that the revised ALE-algorithm overcomes

conceptual problems of former meta-analyses and increases the specificity of the ensuing results without loosing the

sensitivity of the original approach. It may thus provide a methodologically improved tool for coordinate-based

meta-analyses on functional imaging data.

Laird, A.R., Eickhoff, S.B., Li, K., Robin, D.A., Glahn, D.C., Fox, P.T., 2009. Investigating the Functional

Heterogeneity of the Default Mode Network Using Coordinate-Based Meta- Analytic Modeling. J Neurosci.

29, 14496-14505.

The default mode network (DMN) comprises a set of regions that exhibit ongoing, intrinsic activity in the resting

state and task-related decreases in activity across a range of paradigms. However, DMN regions have also been

reported as task-related increases, either independently or coactivated with other regions in the network. Cognitive

subtractions and the use of low-level baseline conditions have generally masked the functional nature of these

regions. Using a combination of activation likelihood estimation, which assesses statistically significant

convergence of neuroimaging results, and tools distributed with the BrainMap database, we identified core regions

in the DMN and examined their functional heterogeneity. Meta-analytic coactivation maps of task-related increases

were independently generated for each region, which included both within-DMN and non-DMN connections. Their

functional properties were assessed using behavioral domain metadata in BrainMap. These results were integrated to

determine a DMN connectivity model that represents the patterns of interactions observed in task-related increases

in activity across diverse tasks. Subnetwork components of this model were identified, and behavioral domain

analysis of these cliques yielded discrete functional properties, demonstrating that components of the DMN are

differentially specialized. Affective and perceptual cliques of the DMN were identified, as well as the cliques

associated with a reduced preference for motor processing. In summary, we used advanced coordinate-based meta-

analysis techniques to explicate behavior and connectivity in the default mode network; future work will involve

applying this analysis strategy to other modes of brain function, such as executive function or sensorimotor systems.

Neumann, J., Fox, P.T., Turner, R., Lohmann, G., 2010. Learning partially directed functional networks

from meta-analysis imaging data. Neuroimage 49 (2), 13721384.

We propose a new exploratory method for the discovery of partially directed functional networks from fMRI

meta-analysis data. The method performs structure learning of Bayesian networks in search of directed probabilistic

dependencies between brain regions. Learning is based on the co-activation of brain regions observed across several

independent imaging experiments. In a series of simulations, we !rst demonstrate the reliability of the method. We

then present the application of our approach in an extensive meta-analysis including several thousand activation

coordinates from more than 500 imaging studies. Results show that our method is able to automatically infer

Bayesian networks that capture both directed and undirected probabilistic dependencies between a number of brain

regions, including regions that are frequently observed in motor-related and cognitive control tasks.

46

Price L, Laird AR, Fox PT. (2009) Modeling Dynamic Functional Neuroimaging Data Using Structural

Equation Modeling. Structural Equation Modeling 16: 147-162.

The aims of this study were to present a method for developing a path analytic network model using data

acquired from positron emission tomography. Regions of interest within the human brain were identified through

quantitative activation likelihood estimation meta-analysis. Using this information, a “true” or population path

model was then developed using Bayesian structural equation modeling. To evaluate the impact of sample size on

parameter estimation bias, proportion of parameter replication coverage, and statistical power, a 2 group

(clinical/control) x 6 (sample size Markov chain Monte Carlo study was conducted. Results indicate that using a

sample size of less than N = 15 per group will produce parameter estimates exhibiting bias greater than 5% and

statistical power below .80.

Robinson, J.L., Laird, A.R., Glahn, D.C., Lovallo, W.R., Fox, P.T., 2010. Metaanalytic connectivity

modeling: delineating the functional connectivity of the human amygdala. Hum. Brain Mapp. 31 (2), 173184.

Functional neuroimaging has evolved into an indispensable tool for noninvasively investigating brain function. A

recent development of such methodology is the creation of connectivity models for brain regions and related

networks, efforts that have been inhibited by notable limitations. We present a new method for ascertaining

functional connectivity of specific brain structures using metaanalytic connectivity modeling (MACM), along with

validation of our method using a nonhuman primate database. Drawing from decades of neuroimaging research and

spanning multiple behavioral domains, the method overcomes many weaknesses of conventional connectivity

analyses and provides a simple, automated alternative to developing accurate and robust models of anatomically-

defined human functional connectivity. Applying MACM to the amygdala, a small structure of the brain with a

complex network of connections, we found high coherence with anatomical studies in nonhuman primates as well as

human-based theoretical models of emotive-cognitive integration, providing evidence for this novel method’s utility.

Grosvald

Grosvald, Gutierrez, & Corina (2011). American Sign Language and gesture processing in deaf signers:

An ERP study. Poster presentation, 18th Annual Meeting of the Cognitive Neuroscience Society, San

Francisco.

Articulatory properties of signed languages share much of the same anatomy and physiology as those involved in

everyday manual actions and gestures. An advance in our understanding of human language would come from a

detailed account of how linguistic and non-linguistic manual actions are differentiated in real time by language users.

To explore this issue, we investigated deaf signers’ brain response when processing ASL sentences incorporating

target items of varying semantic-linguistic status. Each sentence consisted of a frame followed by a last item

belonging to one of four categories: a semantically congruent sign completing the sentence, a semantically

incongruent sign, a non-sign (phonologically legal but non-occurring form), or a non-linguistic grooming action.

We evaluated signers’ response to these final items by examining the N400, a well-studied electrophysiological

(ERP) component known to be sensitive to lexical-semantic integration. This negative-going component is generally

seen at central and parietal scalp sites, peaking about 400 ms after the onset of the relevant stimulus (Kutas &

Hillyard, 1980).

47

Corina, Grosvald et al. (In press). Perceptual invariance or orientation specificity in American Sign

Language? Evidence from repetition priming for signs and gestures. Language and Cognitive Processes.

Repetition priming has been successfully employed to examine stages of processing in a wide variety of

cognitive domains including language, object recognition, and memory. This study uses a novel repetition priming

paradigm in the context of a categorisation task to explore early stages in the processing of American Sign Language

signs and self-grooming gestures. Specifically, we investigated the degree to which deaf signers’ and hearing

nonsigners’ perception of these linguistic or nonlinguistic actions might be differentially robust to changes in

perceptual viewpoint. We conjectured that to the extent that signers were accessing language-specific

representations in their performance of the task, they might show more similar priming effects under different

viewing conditions than hearing subjects. In essence, this would provide evidence for a visually based ‘‘lack of

invariance’’ phenomenon. However, if the early stages of visual-action processing are similar for deaf and hearing

subjects, then no such difference should be found. In both groups, we observed robust effects of viewpoint,

indicating that repetition priming for identical prime_target pairs was greater than in cases of categorisation in

which the prime and target varied in viewpoint. However, we found little evidence of group-related differences that

could be interpreted as effects of perceptual invariance. These outcomes indicate that initial stages of sign and

gesture recognition required for the categorisation of action types do not differ as a function of experience with a

signed language. Instead, our data are consistent with and extend previously described visual-perceptual studies that

have reported evidence for orientation-specific representations of human actions.

Grosvald, Lachaud & Corina (In press). Handshape monitoring: Evaluation of linguistic and perceptual

factors in the processing of American Sign Language. Language and Cognitive Processes.

We investigated the relevance of linguistic and perceptual factors to sign processing by comparing hearing

individuals and deaf signers as they performed a handshape monitoring task, a sign-language analogue to the

phoneme-monitoring paradigms used in many spoken-language studies. Each subject saw a series of brief video

clips, each of which showed either an ASL sign or a phonologically possible but non-lexical “pseudo-sign,” and

responded when the viewed action was formed with a particular handshape. Stimuli varied with respect to the

factors of Lexicality, handshape Markedness (Battison, 1978), and Type, defined according to whether the action is

performed with one or two hands and for two-handed stimuli, whether or not the action is symmetrical. Deaf signers

performed faster and more accurately than hearing non-signers, and effects related to handshape Markedness and

stimulus Type were observed in both groups. However, no effects or interactions related to Lexicality were seen. A

further analysis restricted to the deaf group indicated that these results were not dependent upon subjects’ age of

acquisition of ASL. This work provides new insights into the processes by which the handshape component of sign

forms is recognized in a sign language, the role of language experience, and the extent to which these processes may

or may not be considered specifically linguistic."

Kemmerer

Kemmerer et al. (2010) - The Two-Level Theory of verb meaning: An approach to integrating the

semantics of action with the mirror neuron system

Verbs have two separate levels of meaning. One level reflects the uniqueness of every verb and is called the root.

The other level consists of a more austere representation that is shared by all the verbs in a given class and is called

48

the event structure template. We explore the following hypotheses about how, with specific reference to the motor

features of action verbs, these two distinct levels of semantic representation might correspond to two distinct levels

of the mirror neuron system. Hypothesis 1: Root-level motor features of verb meaning are partially subserved by

somatotopically mapped mirror neurons in the left primary motor and/or premotor cortices. Hypothesis 2: Template-

level motor features of verb meaning are partially subserved by representationally more schematic mirror neurons in

Brodmann area 44 of the left inferior frontal gyrus. Evidence has been accumulating in support of the general

neuroanatomical claims made by these two hypotheses-namely, that each level of verb meaning is associated with

the designated cortical areas. However, as yet no studies have satisfied all the criteria necessary to support the more

specific neurobiological claims made by the two hypotheses-namely, that each level of verb meaning is associated

with mirror neurons in the pertinent brain regions. This would require demonstrating that within those regions the

same neuronal populations are engaged during (a) the linguistic processing of particular motor features of verb

meaning, (b) the execution of actions with the corresponding motor features, and (c) the observation of actions with

the corresponding motor features.

Kemmerer et al. (2008) - Neuroanatomical distribution of five semantic components of verbs: Evidence

from fMRI

The Simulation Framework, also known as the Embodied Cognition Framework, maintains that conceptual

knowledge is grounded in sensorimotor systems. To test several predictions that this theory makes about the neural

substrates of verb meanings; we used functional magnetic resonance imaging (fMRI) to scan subjects' brains while

they made semantic judgments involving five classes of verbs-specifically, Running verbs (e.g., run, jog, walk),

Speaking verbs (e.g., shout, mumble, whisper), Hitting verbs (e.g., hit, poke, jab), Cutting verbs (e.g., cut, slice,

hack), and Change of State verbs (e.g., shatter, smash, crack). These classes were selected because they vary with

respect to the presence or absence of five distinct semantic components-specifically, ACTION; MOTION,

CONTACT, CHANGE OF STATE, and TOOL USE. Based on the Simulation Framework, we hypothesized that

the ACTION component depends on the primary motor and premotor cortices, that the MOTION component

depends on the posterolateral temporal cortex, that the CONTACT component depends on the intraparietal sulcus

and inferior parietal lobule, that the CHANGE OF STATE component depends on the ventral temporal cortex, and

that the TOOL USE component depends on a distributed network of temporal, parietal, and frontal regions. Virtually

all of the predictions were confirmed. Taken together, these findings support the Simulation Framework and extend

our understanding of the neuroanatomical distribution of different aspects of verb meaning.

Kemmerer et al. (2011) - A Functional Role for Motor Simulation in Identifying Tools

Embodied cognition promotes the involvement of the motor system in cognitive processing, such as tool

identification. Although neuropsychological studies suggest that the motor system is not necessary for identifying

tools, it may still have a functional role in tool recognition. To test this possibility, we used a motor interference

task: Participants squeezed a rubber ball in one hand while naming pictures of tools and animals. Participants were

faster and more accurate in naming the tools that were oriented with the handle facing away from the squeezing

hand than in naming the tools that were oriented with the handle facing toward the squeezing hand. There was no

effect of orientation for animals. Given that participants simulate grasping a tool with the hand closest to the handle,

49

this result demonstrates that interfering with the ability to simulate grasping impairs tool naming and suggests that

motor simulation has a functional role in tool identification.

Kemmerer et al. (2010) - Nouns and verbs in the brain: Implications of linguistic typology for cognitive

neuroscience

In recent years, cognitive neuroscience has generated many important findings about how the noun-verb

distinction is implemented in the brain, but at the same time it has largely ignored equally important advances in

linguistic typology concerning the nature of grammatical categories. Following the lead of Evans and Levinson

(2009), we argue that typological data and theory have significant implications not only for the interpretation of

recent neuroscientific discoveries about the noun-verb distinction, but also for the direction of future neuroscientific

research on this topic.

Kemmerer et al. (in press). Behavioral patterns and lesion sites associated with impaired processing of

lexical and conceptual knowledge of actions. Cortex. [Published online.]

To further investigate the neural substrates of lexical and conceptual knowledge of actions, we administered a

battery of six tasks to 226 brain-damaged patients with widely distributed lesions in the left and right cerebral

hemispheres. The tasks probed lexical and conceptual knowledge of actions in a variety of verbal and non-verbal

ways, including naming, word-picture matching, attribute judgments involving both words and pictures, and

associative comparisons involving both words and pictures. Of the 226 patients who were studied, 61 failed one or

more of the six tasks, with four patients being impaired on the entire battery, and varied numbers of patients being

impaired on varied combinations of tasks. Overall, the 61 patients manifested a complex array of associations and

dissociations across the six tasks. The lesion sites of 147 of the 226 patients were also investigated, using formal

methods for lesion-deficit statistical mapping and power analysis of lesion overlap maps. Significant effects for all

six tasks were found in the following left- hemisphere regions: the inferior frontal gyrus; the ventral precentral gyrus,

extending superiorly into what are likely to be hand- related primary motor and premotor areas; and the anterior

insula. In addition, significant effects for 4-5 tasks were found in not only the regions just mentioned, but also in

several other left-hemisphere areas: the ventral postcentral gyrus; the supramarginal gyrus; and the posterior middle

temporal gyrus. These results converge with previous research on the neural underpinnings of action words and

concepts. However, the current study goes considerably beyond most previous investigations by providing extensive

behavioral and lesion data for an unusually large and diverse sample of brain-damaged patients, and by

incorporating multiple measures of verb comprehension. Regarding theoretical implications, the study provides new

support for the Embodied Cognition Framework, which maintains that

conceptual knowledge is grounded in sensorimotor systems.

Kempen

Vosse, Theo & Kempen, Gerard (2009). The Unification Space implemented as a localist neural net:

Predictions and error-tolerance in a constraint-based parser. Cognitive Neurodynamics, 3, 331-346.

We introduce a novel computer implementation of the Unification-Space parser (Vosse and Kempen in Cognition 75:105–143,

2000) in the form of a localist neural network whose dynamics is based on interactive activation and inhibition. The wiring of the

network is determined by Performance Grammar (Kempen and Harbusch in Verb constructions in German and Dutch.

Benjamins, Amsterdam, 2003), a lexicalist formalism with feature unification as binding operation. While the network is

50

processing input word strings incrementally, the evolving shape of parse trees is represented in the form of changing patterns of

activation in nodes that code for syntactic properties of words and phrases, and for the grammatical functions they fulfill. The

system is capable, at least qualitatively and rudimentarily, of simulating several important dynamic aspects of human syntactic

parsing, including garden-path phenomena and reanalysis, effects of complexity (various types of clause embeddings),

faulttolerance in case of unification failures and unknown words, and predictive parsing (expectation-based analysis, surprisal

effects). English is the target language of the parser described.

Vosse, Theo & Kempen, Gerard (2000). Syntactic structure assembly in human parsing: A computational

model based on competitive inhibition and a lexicalist grammar. Cognition, 75, 105-143.

We present the design, implementation and simulation results of a psycholinguistic model of human syntactic processing that

meets major empirical criteria. The parser operates in conjunction with a lexicalist grammar and is driven by syntactic

information associated with heads of phrases. The dynamics of the model are based on competition by lateral inhibition

(‘competitive inhibition’). Input words activate lexical frames (i.e. elementary trees anchored to input words) in the mental

lexicon, and a network of candidate ‘unification links’ is set up between frame nodes. These links represent tentative attachments

that are graded rather than all-or-none. Candidate links that, due to grammatical or ‘treehood’ constraints, are incompatible,

compete for inclusion in the final syntactic tree by sending each other inhibitory signals that reduce the competitor's attachment

strength. The outcome of these local and simultaneous competitions is controlled by dynamic parameters, in particular by the

Entry Activation and the Activation Decay rate of syntactic nodes, and by the Strength and Strength Build-up rate of Unification

links. In case of a successful parse, a single syntactic tree is returned that covers the whole input string and consists of lexical

frames connected by winning Unification links. Simulations are reported of a significant range of psycholinguistic parsing

phenomena in both normal and aphasic speakers of English: (i) various effects of linguistic complexity (single versus double,

center versus right-hand self-embeddings of relative clauses; the difference between relative clauses with subject and object

extraction; the contrast between a complement clause embedded within a relative clause versus a relative clause embedded within

a complement clause); (ii) effects of local and global ambiguity, and of word-class and syntactic ambiguity (including recency

and length effects); (iii) certain difficulty-of-reanalysis effects (contrasts between local ambiguities that are easy to resolve versus

ones that lead to serious garden-path effects); (iv) effects of agrammatism on parsing performance, in particular the performance

of various groups of aphasic patients on several sentence types.

Vosse, Theo & Kempen, Gerard (2009). In defense of competition during syntactic ambiguity resolution.

Journal of Psycholinguistic Research, 38, 1-9

In a recent series of publications (Traxler et al. J Mem Lang 39:558–592, 1998; Van Gompel et al. J Mem Lang 52:284–307,

2005; see also Van Gompel et al. (In: Kennedy, et al.(eds) Reading as a perceptual process, Oxford, Elsevier pp 621–648, 2000);

Van Gompel et al. J Mem Lang 45:225–258, 2001) eye tracking data are reported showing that globally ambiguous (GA)

sentences are read faster than locally ambiguous (LA) counterparts. They argue that these data rule out constraint-based models

where syntactic and conceptual processors operate concurrently and syntactic ambiguity resolution is accomplished by

competition. Such models predict the opposite pattern of reading times. However, this argument against competition is valid only

in conjunction with two standard assumptions in current constraint-based models of sentence comprehension: (1) that syntactic

competitions (e.g., Which is the best attachment site of the incoming constituent?) are pooled together with conceptual

competitions (e.g., Which attachment site entails the most plausible meaning?), and (2) that the duration of a competition is a

function of the overall (pooled) quality score obtained by each competitor. We argue that it is not necessary to abandon

competition as a successful basis for explaining parsing phenomena and that the above-mentioned reading time data can be

accounted for by a parallel-interactive model with conceptual and syntactic processors that do not pool their quality scores

together. Within the individual linguistic modules, decision-making can very well be competition-based.

51

Snijders, Tineke M.; Vosse, Theo; Kempen, Gerard; Van Berkum Jos J.A.; Petersson, Karl Magnus &

Hagoort, Peter (2009). Retrieval and unification of syntactic structure in sentence comprehension: an fMRI

study using word-category ambiguity. Cerebral Cortex, 19, 1493-1503.

Sentence comprehension requires the retrieval of single word information from long-term memory, and the integration of this

information into multiword representations. The current functional magnetic resonance imaging study explored the hypothesis

that the left posterior temporal gyrus supports the retrieval of lexicalsyntactic information, whereas left inferior frontal gyrus

(LIFG) contributes to syntactic unification. Twenty-eight subjects read sentences and word sequences containing word-category

(noun-verb) ambiguous words at critical positions. Regions contributing to the syntactic unification process should show

enhanced activation for sentences compared to words, and only within sentences display a larger signal for ambiguous than

unambiguous conditions. The posterior LIFG showed exactly this predicted pattern, confirming our hypothesis that LIFG

contributes to syntactic unification. The left posterior middle temporal gyrus was activated more for ambiguous than

unambiguous conditions (main effect over both sentences and word sequences), as predicted for regions subserving the retrieval

of lexical-syntactic information from memory. We conclude that understanding language involves the dynamic interplay between

left inferior frontal and left posterior temporal regions.

Harbusch, Karin & Kempen, Gerard (2002). A quantitative model of word order and movement in

English, Dutch, and German complement constructions. In Proceedings of the 19th International Conference

on Computational Linguistics (COLING-2002), Taipei (Taiwan). San Francisco: Morgan Kaufmann. [pp.

328-334]

We present a quantitative model of word order and movement constraints that enables a simple and uniform

treatment of a seemingly heterogeneous collection of linear order phenomena in English, Dutch and German

complement constructions (Wh-extraction, clause union, extraposition, verb clustering, particle movement, etc.).

Underlying the scheme are central assumptions of the psycholinguistically motivated Performance Grammar (PG).

Here we describe this formalism in declarative terms based on typed feature unification. PG allows a homogenous

treatment of both the within- and between-language variations of the ordering phenomena under discussion, which

reduce to different settings of a small number of quantitative parameters.

Kempen, Gerard; Olsthoorn, Nomi & Sprenger, Simone (in press, 2011). Grammatical workspace sharing

during language production and language comprehension: Evidence from grammatical multitasking.

Language and Cognitive Processes.

Grammatical encoding and grammatical decoding (in sentence production and comprehension, respectively) are

often portrayed as independent modalities of grammatical performance that only share declarative resources: lexicon

and grammar. The processing resources subserving these modalities are supposed to be distinct. In particular, one

assumes the existence of two workspaces where grammatical structures are assembled and temporarily maintained—

one for each modality. An alternative theory holds that the two modalities share many of their processing resources

and postulates a single mechanism for the online assemblage and short-term storage of grammatical structures: a

shared workspace. We report two experiments with a novel “grammatical multitasking” paradigm: The participants

had to read (i.e. decode) and to paraphrase (encode) sentences presented in fragments, responding to each input

fragment as fast as possible with a fragment of the paraphrase. The main finding was that grammatical constraints

with respect to upcoming input that emanate from decoded sentence fragments are immediately replaced by

grammatical expectations emanating from the structure of the corresponding paraphrase fragments. This evidences

52

that the two modalities have direct access to, and operate upon, the same (i.e. token-identical) grammatical

structures. This is possible only if the grammatical encoding and decoding processes command the same, shared

grammatical workspace. Theoretical implications for important forms of grammatical multitasking—self-monitoring,

turn-taking in dialogue, speech shadowing, and simultaneous translation—are explored.

Vosse, Theo & Kempen, Gerard (2008). Parsing verb-final clauses in German: Garden- path and ERP

effects modeled by a parallel dynamic parser. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings

of the 30th Annual Conference of the Cognitive Science Society (Washington DC, July 2008). Austin, TX:

Cognitive Science Society. [pp.261-266].

Experimental sentence comprehension studies have shown that superficially similar German clauses with verb-

final word order elicit very different garden-path and ERP effects. We show that a computer implementation of the

Unification Space parser (Vosse & Kempen, 2000) in the form of a localist-connectionist network can model the

observed differences, at least qualitatively. The model embodies a parallel dynamic parser that, in contrast with

existing models, does not distinguish between consecutive first-pass and reanalysis stages, and does not use

semantic or thematic roles. It does use structural frequency data and animacy information.

Lee

Arbib, M. A. and J. Lee (2007). Vision and Action in the Language-Ready Brain: From Mirror Neurons to

SemRep. BVAI 2007 (Brain Vision & Artificial Intelligence, 2007), LNCS 4729. F. Mele. Berlin/Heidelberg,

Springer-Verlag: 104-123.

The general setting for our work is to locate language perception and production within the broader context of

brain mechanisms for action and perception in general, modeling brain function in terms of the competition and

cooperation of schemas. Particular emphasis is placed on mirror neurons – neurons active both for execution of a

certain class of actions and for recognition of a (possibly broader) class of similar actions. We build on the early

VISIONS model of schema-based computer analysis of static scenes to present SemRep, a graphical representation

of dynamic visual scenes designed to support the generation of varied descriptions of episodes. Mechanisms for

parsing and production of sentences are currently being implemented within Template Construction Grammar

(TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics.

Arbib, M. A. and J. Lee (2008). Describing visual scenes: Towards a neurolinguistics based on construction

grammar. Brain Research 1225: 146-162.

The present paper is part of a larger effort to locate the production and perception of language within the broader

context of brain mechanisms for action and perception more generally. Here we model function in terms of the

competition and cooperation of schemas. We use the task of describing visual scenes to explore the suitability of

Construction Grammar as an appropriate framework for a schema-based linguistics. We recall the early VISIONS

model of schema-based computer analysis of static visual scenes and then introduce SemRep as a graphical

representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes. We

report preliminary results on implementing the production of sentences using Template Construction Grammar

(TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics. We

summarize data on neural correlates relevant to future work on TCG within the context of neurolinguistics, and

show how the relation between SemRep and TCG can serve as the basis for modeling language comprehension.

53

MacWhinney

MacWhinney, B. (2005). A Unified Model of Language Acquisition. In J. Kroll & A. De Groot (Eds.)

Handbook of bilingualism: Psycholinguistic approaches. Oxford University Press.

This paper presents an extended formulation of the Competition Model. The extended model is designed to account for a

larger range of phenomena in first and second language acquisition, including bilingualism. As in the classic version of the

Competition Model, competition is at the core of a set of non-modular interacting forces. However, now the various inputs to

competition are described in terms of six additional subcomponents: arenas, cues, chunking, storage, codes, and resonance.

Learning is viewed as a resonant process that relies on storage, chunking, and support to acquire new mappings.

Hernandez, A., Li, P., & MacWhinney, B. (2005). The emergence of competing modules in bilingualism.

Trends in Cognitive Sciences, 9, 220-225.

How does the brain manage to store and process multiple languages without encountering massive interference and transfer?

Unless we believe that bilinguals live in two totally unconnected cognitive worlds, we would expect far more transfer than

actually occurs. However, imaging and lesion studies have not provided consistent evidence for the strict neuronal separation

predicted by the theory of modularity. We suggest that emergentist theory offers a promising alternative. It emphasizes the

competitive interplay between multiple languages during childhood and by focusing on the dual action of competition and

entrenchment, avoids the need to invoke a critical period to account for age of acquisition effects in second-language learning.

This view instantiates the motto formulated by Elizabeth Bates that‘modules are made, not born.’

MacWhinnney, B. (2005). The emergence of linguistic form in time. Connection Science, 17, 191-211.

Linguistic forms are shaped by forces operating on vastly different time scales. Some of these forces operate directly at the

moment of speaking, whereas others accumulate over time in personal and social memory. Our challenge is to understand how

forces with very different time scales mesh together in the current moment to determine the emergence of linguistic form.

MacWhinney, B. (2008). How mental models encode embodied linguistic perspectives. In Klatzky, R.,

MacWhinney, B., and Behrmann, M. (Eds.). Embodiment, Ego-space, adn Action (pp. 365-405). Lawrence

Erlbaum.

Humans demonstrate a remarkable ability to take other people’s perspectives. When we watch movies, we find ourselves

identifying with the actors, sensing their joys, hopes, fears, and sorrows. As viewers, we can be moved to exhilaration as we

watch our heroes overcome obstacles; or we can be moved to tears when they suffer losses and defeats. This process of

identification does not always have to be linked to intense emotional involvement. At a soccer match, we can follow the

movements of a player moving in to shoot for a goal. We can identify with the player’s position, stance, and maneuvers against

the challenges offered by the defenders. We can track the actions, as the player drives toward the goal and kicks the ball into the

net. This ability to take the perspective of another person is very general. Just as we follow the movements of dancers, actors, and

athletes, we can also follow the thoughts and emotions expressed by others in language. In this paper, we will explore the ways in

which language builds upon our basic system for projecting the body image to support a rich system of perspective tracking and

mental model construction.

Li, P., Farkas, I., & MacWhinney, B. (2004). Early lexical development in a self-organizing neural

network. Neural Networks 17, 1345-1362.

In this paper we present a self-organizing neural network model of early lexical development called DevLex. The network

consists of two self-organizing maps (a growing semantic map and a growing phonological map) that are connected via

associative links trained by Hebbian learning. The model captures a number of important phenomena that occur in early lexical

acquisition by children, as it allows for the representation of a dynamically changing linguistic environment in language learning.

In our simulations, DevLex develops topographically organized representations for linguistic categories over time, models lexical

54

confusion as a function of word density and semantic similarity, and shows age-of-acquisition effects in the course of learning a

growing lexicon. These results match up with patterns from empirical research on lexical development, and have significant

implications for models of language acquisition based on self-organizing neural networks.

Li, P., Zhao, X., and MacWhinney, B. (2007) Dynamic Self-Organization and Early Lexical Development

in Children, Cognitive Science 31 581–612

In this study we present a self-organizing connectionist model of early lexical development.We call this model

DevLex-II, based on the earlier DevLex model. DevLex-II can simulate a variety of empirical patterns in children’s

acquisition of words. These include a clear vocabulary spurt, effects of word frequency and length on age of

acquisition, and individual differences as a function of phonological short-term memory and associative capacity.

Further results from lesioned models indicate developmental plasticity in the network’s recovery from damage, in a

non-monotonic fashion. We attribute the network’s abilities in accounting for lexical development to interactive

dynamics in the learning process. In particular, variations displayed by the model in the rate and size of early

vocabulary development are modulated by (a) input characteristics, such as word frequency and word length, (b)

consolidation of lexical-semantic representation, meaning-form association, and phonological short-term memory,

and (c) delayed processes due to interactions among timing, severity, and recoverability of lesion. Together, DevLex

and DevLex-II provide an accurate computational account of early lexical development.

MacWhinney, B. Item-based Patterns in Early Syntactic Development

This paper presents an account of first language acquisition based on the child�

s learning of item-based patterns

(IBPs). These patterns involve grammatical dependencies between a lexical predicate (such as more) and its

arguments (such as milk) to form a new cluster (such as more milk). Children can use simple, systematic inductive

operations to acquire these positional patterns, and then to generalize them into fuller feature-based and global

constructions. Together, these patterns can provide a full account of the learning of syntax. Recent work uses this

framework to construct computational simulations of children�

s syntactic development that match up well with

extensive, publicly available, corpus data available from the CHILDES database.

MacWhinney, B. The Logic of the Unified Model, In S. Gass and A. Mackey (Ed.) Handbook of Second

Language Acquisition – Routledge.

Many people believe that learning a second language is fundamentally different from learning a first language.

Evidence of this fundamental difference comes from the fact that first language acquisition almost invariably

produces full native speaker competence, whereas many second language learners achieve only partial success in

learning their new language. Some researchers believe that this difference in levels of ultimate attainment result

arises because, after the expiration of a certain critical period, the learning mechanisms that subserve first language

learning atrophy or expire. The Unified Competition Model (MacWhinney, 2008b) takes a different approach to this

issue. Instead of attributing differences between first and second language learning to the effects of a critical period,

these differences are attributed to the differential interplay between riskgenerating processes and protective, support

processes. For L1 learning, the five risk factors are entrenchment, parasitism, misconnection, negative transfer, and

isolation. To overcome these five risk factors, adults can rely on the support processes of resonance, internalization,

chunking, positive transfer, and participation. All of these risk factors and support processes are available to children,

as well as adults. What differs between L1 and L2 learning is the way in which these processes are configured.

55

Miikkulainen

Grasemann, U., Hoffman, R., and Miikkulainen, R. (in press). Modeling Acute and Compensated

Language Disturbance in Schizophrenia. In Proceedings of the 33rd Annual Meeting of the Cognitive Science

Society.

http:/nn.cs.utexas.edu/?grasemann:cogsci11

No current laboratory test can reliably identify patients with schizophrenia. Instead, key symptoms are observed

via language, including derailments, where patients cannot follow a coherent storyline, and delusions, where false

beliefs are repeated as fact. Brain processes underlying these and other symptoms remain unclear, and characterizing

them would greatly enhance our understanding of schizophrenia. In this situation, computational models can be

valuable tools to formulate testable hypotheses and to complement clinical research. This work aims to capture the

link between biology and schizophrenic symptoms using DISCERN, a connectionist model of human story

processing. Competing illness mechanisms proposed to underlie schizophrenia are simulated in DISCERN, and are

evaluated at the level of narrative language, i.e. the same level used to diagnose patients. The result is the first

simulation of abnormal storytelling in schizophrenia, both in acute psychotic and compensated stages of the

disorder. Of all illness models tested, hyperlearning, a model of overly intense memory consolidation, produced the

best fit to the language abnormalities of stable outpatients, as well as compelling models of acute psychotic

symptoms. If validated experimentally, the hyperlearning hypothesis could advance the current understanding of

schizophrenia, and provide a platform for developing future treatments for this disorder.

Hoffman, R.E., Grasemann, U., Gueorguieva, R., Quinlan, D., Lane, D., and Miikkulainen, R. (2011).

Using Computational Patients to Evaluate Illness Mechanisms in Schizophrenia, Biological Psychiatry.

http://nn.cs.utexas.edu/?grasemann:biopsych11

Background: Various malfunctions involving working memory, semantics, prediction error, and dopamine

neuromodulation have been hypothesized to cause disorganized speech and delusions in schizophrenia.

Computational models may provide insights into why some mechanisms are unlikely, suggest alternative

mechanisms, and tie together explanations of seemingly disparate symptoms and experimental findings.

Methods: Eight corresponding illness mechanisms were simulated in DISCERN, an artificial neural network

model of narrative understanding and recall. For this study, DISCERN learned sets of autobiographical and

impersonal crime stories with associated emotion coding. In addition, 20 healthy control subjects and 37 patients

with schizophrenia or schizoaffective disorder matched for age, gender, and parental education were studied using a

delayed story recall task. A goodness-of-fit analysis was performed to determine the mechanism best reproducing

narrative breakdown profiles generated by healthy control subjects and patients with schizophrenia. Evidence of

delusion-like narratives was sought in simulations best matching the narrative breakdown profile of patients.

Results: All mechanisms were equivalent in matching the narrative breakdown profile of healthy control subjects.

However, exaggerated prediction-error signaling during consolidation of episodic memories, termed hyperlearning,

was statistically superior to other mechanisms in matching the narrative breakdown profile of patients. These

simulations also systematically confused autobiographical agents with impersonal crime story agents to model fixed,

self-referential delusions.

56

Conclusions: Findings suggest that exaggerated prediction-error signaling in schizophrenia intermingles and

corrupts narrative memories when incorporated into long-term storage, thereby disrupting narrative language and

producing fixed delusional narratives. If further validated by clinical studies, these computational patients could

provide a platform for developing and testing novel treatments.

Miikkulainen, R., Kiran, S. (2009). Modeling the Bilingual Lexicon of an Individual Subject, In

Proceedings of the Workshop on Self-Organizing Maps (WSOM'09), Berlin, Springer Lexicon is a central component in any language processing system, whether human or artificial. Recent empirical evidence

suggests that a multilingual lexicon consists of a single component representing word meanings, and separate component for the

symbols in each language. These components can be modeled as self-organizing maps, with associative connections between

them implementing comprehension and production. Computational experiments in this paper show that such a model can trained

to match the proficiency and age of acquisition of particular bilingual individuals. In the future, it may be possible to use such

models to predict the effect of rehabilitation of bilingual aphasia, resulting in more effective treatments.

Grasemann, U., Sandberg, C., Kiran, S., and Miikkulainen, R. (2011). Impairment and Rehabilitation in

Bilingual Aphasia: A SOM-Based Model. In \Proceedings of the Eighth Workshop on Self-Organizing Maps

(WSOM'2011, Espoo, Finland).

http://nn.cs.utexas.edu/?grasemann:wsom11

ilingual aphasia is of increasing interest because a large and growing proportion of the world's population is

bilingual. Current clinical research on this topic cannot provide specific recommendations on which language

treatment should focus in a bilingual aphasic individual and to what extent cross-language transfer occurs during or

after rehabilitation. This paper describes a SOM-based model of the bilingual lexicon, and reports on simulations of

impairment and rehabilitation in bilingual aphasia. The goal is to create computational methods that can complement

clinical research in developing a better understanding of mechanisms underlying recovery, and that could be used in

the future to predict the most beneficial treatment for individual patients.

Williams, P., Miikkulainen, R. (2006). Grounding Language in Descriptions of Scenes, In Proceedings of

the 28th Annual Meeting of the Cognitive Science Society

The problem of how abstract symbols, such as those in systems of natural language, may be grounded in perceptual

information presents a significant challenge to several areas of research. This paper presents the GLIDES model, a neural

network architecture that shows how this symbol-grounding problem can be solved through learned relationships between simple

visual scenes and linguistic descriptions. Unlike previous models of symbol grounding, the model's learning is completely

unsupervised, utilizing the principles of self organization and Hebbian learning and allowing direct visualization of how concepts

are formed and grounding occurs. Two sets of experiments were conducted to evaluate the model. In the first set, linguistic test

stimuli were presented and the scenes that were generated by the model were evaluated as the grounding of the language. In the

second set, the model was presented with visual test samples and its language generation capabilities based on the grounded

representations were assessed. The results demonstrate that symbols can be grounded based on associations of perceptual and

linguistic representations, and the grounding can be made transparent. This transparency leads to unique insights into symbol

grounding, including how many-to-many mappings between symbols and referents can be maintained and how concepts can be

formed from cooccurrence relationships.

Nielsen

The Brede database: a small database for functional neuroimaging

http://www2.imm.dtu.dk/~fn/Nielsen2003Brede_abstract/Nielsen2003Brede_abstract.html

57

Abstract: Introduction -- We describe the Brede neuroinformatics database that provides data for novel

information retrieval techniques and automated meta-analyses. Data -- The database is inspired by the hierarchical

structure of BrainMap [1] with scientific articles (bib structures) on the highest level containing one or more

experiments (exp structure, corresponding to a contrast in general linear model analyses), these in turn comprising

one or more locations (loc structures). The information on the bib level (author, title, ...) is setup automatically

from PubMed while the rest of the information is entered manually in a Matlab graphical user interface. On the loc

level this includes the 3D stereotactic coordinates in either Talairach or MNI space, the brain area (functional,

anatomical or cytoarchitectonic area) and magnitude values such as Z-score and P-value. On the exp level

information such as modality, scanner and behavioral domain are recorded with external components (such as face

recognition or kinetic boundaries) organized in a directed graph and marked up with Medical Subject Headings

(MeSH) where possible. Items in the database are identified with unique numbers and the type of identifier is given

a unique string, e.g., WOBIB: 27 for an Epstein and Kanwisher paper. This will allow Internet search engine to

identify the phrase. For storing the data we employ a simple XML format that we denote poor-man’s XML (pXML)

with no attributes and no empty tags. The database presently consists of data constructed from 40 scientific

articles, containing 134 experiments and 882 locations...

Mining for associations between text and brain activation in a functional neuroimaging database

http://www.springerlink.com/content/m01l058504070750/

Abstract: We describe a method for mining a neuroimaging database for associations between text and brain

locations. The objective is to discover association rules between words indicative of cognitive function as described

in abstracts of neuroscience papers and sets of reported stereotactic Talairach coordinates. We invoke a simple

probabilistic framework in which kernel density estimates are used to model distributions of brain activation foci

conditioned on words in a given abstract. The principal associations are found in the joint probability density

between words and voxels. We show that the statistically motivated associations are well aligned with general

neuroscientific knowledge.

Modeling of activation data in the BrainMapTM database: Detection of outliers

www2.imm.dtu.dk/~fn/ps/Nielsen2000Modeling_text.ps

Abstract: We describe a system for meta-analytical modeling of activation foci from functional neuroimaging

studies. Our main vehicle is a set of density models in Talairach space capturing the distribution of activation foci in

sets of experiments labeled by lobar anatomy. One important use of such density models is identification of novelty,

i.e., low probability database events. We rank the novelty of the outliers and investigate the cause for 21 of the most

novel, finding several outliers that are entry and transcription errors or infrequent or non-conforming terminology.

We briefly discuss the use of atlases for outlier detection.

Lost in localization: A solution with neuroinformatics 2.0?

http://onlinelibrary.wiley.com/doi/10.1002/hbm.10012/abstract

Abstract: The commentary by Derrfuss and Mar (Derrfuss, J., Mar, R.A., 2009. Lost in localization: The need for

a universal coordinate database. NeuroImage, doi:10.1016/j.neuroimage.2009.01.053.) discusses some of the

limitations of the present databases and calls for a universal coordinate database. Here I discuss further issues and

propose another angle to the solution of a universal coordinate database with the use of wiki technology.

58

Visualizing data mining results with the Brede tools

http://www.frontiersin.org/neuroinformatics/10.3389/neuro.11/026.2009/abstract

A few neuroinformatics databases now exist that record results from neuroimaging studies in the form of brain

coordinates in stereotaxic space. The Brede Toolbox was originally developed to extract, analyze and visualize data

from one of them the BrainMap database. Since then the Brede Toolbox has expanded and now includes its own

database with coordinates along with ontologies for brain regions and functions: The Brede Database. With Brede

Toolbox and Database combined, we setup automated workflows for extraction of data, mass meta-analytic data

mining and visualizations. Most of the Web presence of the Brede Database is established by a single script

executing a workflow involving these steps together with a final generation of Web pages with embedded

visualizations and links to interactive three-dimensional models in the Virtual Reality Modeling Language. Apart

from the Brede tools I briefly review alternate visualization tools and methods for Internet-based visualization and

information visualization as well as portals for visualization tools.

Oztop

All can be downloaded from http://www.cns.atr.jp/~erhan/ErhanOztop_publications.html:

Chaminade T, Oztop E, Cheng G, Kawato M. From self-observation to imitation: visuomotor association

on a robotic hand. Brain Res Bull. 2008 Apr 15;75(6):775-84. Epub 2008 Feb 14.

Being at the crux of human cognition and behaviour, imitation has become the target of investigations ranging

from experimental psychology and neurophysiology to computational sciences and robotics. It is often assumed that

the imitation is innate, but it has more recently been argued, both theoretically and experimentally, that basic forms

of imitation could emerge as a result of self-observation. Here, we tested this proposal on a realistic experimental

platform, comprising an associative network linking a 16 degrees of freedom robotic hand and a simple visual

system. We report that this minimal visuomotor association is sufficient to bootstrap basic imitation. Our results

indicate that crucial features of human imitation, such as generalization to new actions, may emerge from a

connectionist associative network. Therefore, we suggest that a behaviour as complex as imitation could be, at the

neuronal level, founded on basic mechanisms of associative learning, a notion supported by a recent proposal on the

developmental origin of mirror neurons. Our approach can be applied to the development of realistic cognitive

architectures for humanoid robots as well as to shed new light on the cognitive processes at play in early human

cognitive development.

Oztop, E, Imamizu, H, Cheng, G, Kawato, M (2006) A computation model of anterior intraparietal (AIP)

neurons. Neurocomputing, 69(10-12): 1354-1361.

The monkey parietal anterior intraparietal area (AIP) is part of the grasp planning and execution circuit which

contains neurons that encode object features relevant for grasping, such as the width and the height. In this study we

focus on the formation of AIP neurons during grasp development. We propose and implement a neural network

structure and a learning mechanism that is driven by successful grasp experiences during early grasp development.

The simulations show that learning leads to emergence of units that have similar response properties as the AIP

visual-dominant neurons. The results may have certain implications for the function of AIP neurons and thus should

stimulate new experiments that cannot only verify/falsify the model but also advance our understanding of the

visuomotor learning mechanisms employed by the primate brain.

59

Oztop E, Kawato M, Arbib M (2006) Mirror Neurons and Imitation: A Computationally Guided Review.

Neural Networks 19: 254-271

Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals

the presence of ‘mirror systems’ (not individual neurons) in the human. Current conceptual models attribute high

level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of

these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on

mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of

current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror

systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular

selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often

overlook the computational requirements for posited functions, while too many computational models adopt the

erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines

the gap between conceptual and computational models and points out the research effort required from both sides to

reduce this gap. q 2006 Published by Elsevier Ltd. Keywords: Mirror neuron; Action understanding; Imitation;

Language; Computational model

Oztop E, Wolpert D, Kawato M (2005) Mental state inference using visual control parameters. Cogn Brain

Res 22: 129-151

Although we can often infer the mental states of others by observing their actions, there are currently no

computational models of this remarkable ability. Here we develop a computational model of mental state inference

that builds upon a generic visuomanual feedback controller, and implements mental simulation and mental state

inference functions using circuitry that subserves sensorimotor control. Our goal is (1) to show that control

mechanisms developed for manual manipulation are readily endowed with visual and predictive processing

capabilities and thus allows a natural extension to the understanding of movements performed by others; and (2) to

give an explanation on how cortical regions, in particular the parietal and premotor cortices, may be involved in

such dual mechanism. To analyze the model, we simulate tasks in which an observer watches an actor performing

either a reaching or a grasping movement. The observer’s goal is to estimate the dmental stateT of the actor: the goal

of the reaching movement or the intention of the agent performing the grasping movement. We show that the motor

modules of the observer can be used in a dsimulation modeT to infer the mental state of the actor. The simulations

with different grasping and non-straight line reaching strategies show that the mental state inference model is

applicable to complex movements. Moreover, we simulate deceptive reaching, where an actor imposes false beliefs

about his own mental state on an observer. The simulations show that computational elements developed for

sensorimotor control are effective in inferring the mental states of others. The parallels between the model and

cortical organization of movement suggest that primates might have developed a similar resource utilization strategy

for action understanding, and thus lead to testable predictions about the brain mechanisms of mental state inference.

Oztop E., Arbib M.A. (2002) Schema Design and Implementation of the Grasp-Related Mirror Neuron

System. Biological Cybernetics 87: (2) 116-140

Mirror neurons within a monkey’s premotor area F5 fire not only when the monkey performs a certain class of

actions but also when the monkey observes another monkey (or the experimenter) perform a similar action. It has

60

thus been argued that these neurons are crucial for understanding of actions by others. We o�er the hand-state

hypothesis as a new explanation of the evolution of this capability: the basic functionality of the F5 mirror system is

to elaborate the appropriate feedback – what we call the hand state – for opposition-space based control of manual

grasping of an object. Given this functionality, the social role of the F5 mirror system in understanding the actions of

others may be seen as an exaptation gained by generalizing from one’s own hand to an other’s hand. In other words,

mirror neurons first evolvedto augment the ‘‘canonical’’ F5 neurons (active during self-movement based on

observation of an object) by providing visual feedback on ‘‘handstate,’’ relating the shape of the hand to the shape

of the object. We then introduce the MNS1 (mirror neuron system 1) model of F5 and related brain regions. The

existing Fagg–Arbib–Rizzolatti–Sakata model represents circuitry for visually guided grasping of objects, linking

the anterior intraparietal area (AIP) with F5 canonical neurons. The MNS1 model extends the AIP visual pathway

by also modeling pathways, directed toward F5 mirror neurons, which match arm– hand trajectories to the

a�ordances and location of a potential target object. We present the basic schemas for the MNS1 model, then

aggregate them into three ‘‘grand schemas’’ – visual analysis of hand state, reach and grasp, andthe core mirror

circuit – for each of which we present a useful implementation (a non-neural visual processing system, a multijoint

3-D kinematics simulator, and a learning neural network, respectively). With this implementation we show how the

mirror system may learn to recognize actions already in the repertoire of the F5 canonical neurons. We show that the

connectivity pattern of mirror neuron circuitry can be estab- lishedthrough training, andthat the resultant network

can exhibit a range of novel, physiologically interesting behaviors during the process of action recognition. We train

the system on the basis of final grasp but then observe the whole time course of mirror neuron activity, yielding

predictions for neurophysiological experiments under conditions of spatial perturbation, altered kinematics,

andambiguous grasp execution which highlight the importance of the timing of mirror neuron activity.

Ugur E, Oztop E, Sahin E (2011) Goal emulation and planning in perceptual space using learned

affordances. Robotics and Autonomous Systems 59, 580-595

In this paper, we show that through self-interaction and self-observation, an anthropomorphic robot equipped

with a range camera can learn object affordances and use this knowledge for planning. In the first step of learning,

the robot discovers commonalities in its action-effect experiences by discovering effect categories. Once the effect

categories are discovered, in the second step, affordance predictors for each behavior are obtained by learning the

mapping from the object features to the effect categories. After learning, the robot can make plans to achieve desired

goals, emulate end states of demonstrated actions, monitor the plan execution and take corrective actions using the

perceptual structures employed or discovered during learning. We argue that the learning system proposed shares

crucial elements with the development of infants of 7–10 months age, who explore the environment and learn the

dynamics of the objects through goal-free exploration. In addition, we discuss goal emulation and planning in

relation to older infants with no symbolic inference capability and non-linguistic animals which utilize object

affordances to make action plans.

Schilling

Schilling, M., Cruse, H.: Cognition as recruitment of reactive systems. (submitted,

http://www.icsi.berkeley.edu/~lucag/archive/manuscript_schilling_cruse%20Reactive%20Based%20Cognition%20

AfterSubm%20Layouted.pdf ).

61

It has often been stated that a neuronal system to become a cognitive one has to be complex enough, complexity

often paralleled with size. In contrast, we argue that basic requirements can be fulfilled already by small neuronal

systems raising the question what are the crucial neuronal structures and processes that make a system a cognitive

one. Our main claim is that it is not possible to understand cognition without taking into account the reactive basis of

an embedded cognitive system. Therefore, we propose a network, termed reaCog, that, first, is able to deal with a

specific domain of behavior (six-legged-walking). Second, we show how this system can be expanded to be able to

plan ahead and invent behaviors in order to solve new problems. The expansion consists of a manipulable body

model and a WTA network that governs the activation of the behaviors used for planning ahead

Schilling, M. (2011): Universally manipulable body models—dual quaternion representations in layered

and dynamic MMCs. Autonomous Robots 30:4, 399–425; http://dx.doi.org/10.1007/s10514-011-9226-3

Surprisingly complex tasks can be solved using a behaviour-based, reactive control system, i.e., a system that

operates without an explicit internal representation of the environment and the own body. Nevertheless, application

of internal representations has gained interest in recent years because such internal representations can be used to

solve problems of perception and motor control (sensor fusion, inverse modeling) and may in addition be applied to

higher cognitive functions as are the ability to plan ahead. To endow such a system with the ability to find new

behavioural solutions to a given problem in a broad range of possibilities, the internal representation must be

universally manipulable, i.e. the model should be able to simulate all movements that are physically possible for the

body given. Using recurrent neural networks, models showing this faculty have been proposed being based on the

principle of mean of multiple computation (MMC). The extension of this approach to three dimensions requires the

introduction of a joint angle representation which allows for computation of mean values. Here we use dual

quaternions that are singularity-free and unambiguous which allow for shortest path interpolation. In addition, it has

been shown that dual quaternions are the most efficient and most compact form for representing rigid

transformations. The model can easily be adapted to bodies of arbitrary geometries. The extended MMC net

introduced in this article represents a holistic system that can—following the principle of pattern completion—

likewise be used as an inverse model, a forward model, for sensor fusion or other, related capabilities.

Schilling, M. (in press): Learning by seeing—associative learning of visual features through mental

simulation of observed action. ECAL 2011, Paris; http://www.icsi.berkeley.edu/~mschilli/schilling2011ecal.pdf

Internal representations employed in cognitive tasks have to be embodied. The flexible use of such grounded

models allows for higher-level function like planning ahead, cooperation and communication. But at the same time

this flexibility presupposes that the utilized internal models are interrelating multiple modalities. In this article we

present how an internal body model serving motor control tasks can be recruited for learning to recognize

movements performed by another agent. We show that—as the movements are governed by an equal underlying

internal model—it is sufficient to observe the other agent performing a series of movements and that there is no

supervised learning necessary, i.e. the learning agent does not require access to the performing agents postural

information (joint configurations). Instead, through the shared underlying dynamics the mapping can be

bootstrapped by the observing agent from the sequence of visual input features.

Small

Small et al. (2011) - From Language Comprehension to Action Understanding and Back Again

62

A controversial question in cognitive neuroscience is whether comprehension of words and sentences engages

brain mechanisms specific for decoding linguistic meaning or whether language comprehension occurs through

more domain-general sensorimotor processes. Accumulating behavioral and neuroimaging evidence suggests a role

for cortical motor and premotor areas in passive action-related language tasks, regions that are known to be involved

in action execution and observation. To examine the involvement of these brain regions in language and

nonlanguage tasks, we used functional magnetic resonance imaging (fMRI) on a group of 21 healthy adults. During

the fMRI session, all participants 1) watched short object-related action movies, 2) looked at pictures of man-made

objects, and 3) listened to and produced short sentences describing object-related actions and man-made objects. Our

results are among the first to reveal, in the human brain, a functional specialization within the ventral premotor

cortex (PMv) for observing actions and for observing objects, and a different organization for processing sentences

describing actions and objects. These findings argue against the strongest version of the simulation theory for the

processing of action-related language.

Small et al. (2011) - Interpretation-mediated changes in neural activity during language comprehension

Using functional magnetic resonance imaging (fMRI), we identified cortical regions mediating interpretive

processes that take place during language comprehension. We manipulated participants' interpretation of texts by

asking them to focus on action-, space-, or time-related features while listening to identical short stories. We identify

several cortical regions where activity varied significantly in response to this attention manipulation, even though

the content being processed was exactly the same. Activity in the posterior and anterior sections of the left inferior

frontal gyrus (IFG), which are thought to have different sensitivities to high-level language processing, was

modulated by the listeners' attentional focus, but in ways that were quite different. The posterior left IFG (Pars

Opercularis) showed different activity levels for the three conditions. However, a population coding analysis

demonstrated similar distributions of activity across conditions. This suggests that while the gain of the response in

the Pars Opercularis was modulated, its core organization was relatively invariant across the experimental

conditions. In the anterior left IFG (Pars Triangularis), the analysis of population codes revealed different activity

patterns between conditions: there was little similarity between activity during time-attention and action- and space-

attention, however there were similar activity patterns while attending to space and action information. In addition,

both the left superior temporal gyrus and sulcus showed greater activity in the space and action attention conditions

when contrasted with time attention. We discuss these findings in light of work on the role of left IFG in processing

semantic information in language, and in light of theories suggesting that temporal information in language is

processed in the brain using similar mechanisms as spatial information. Our findings suggest that a substantial

source of variance in neural activity during language comprehension emerges from the internally-driven,

information-seeking preferences of listeners rather than the syntactic or semantic properties of a text.

Small et al. (2009) - Gestures Orchestrate Brain Networks for Language Understanding

Although the linguistic structure of speech provides valuable communicative information, nonverbal behaviors

can offer additional, often disambiguating cues. In particular, being able to see the face and hand movements of a

speaker facilitates language comprehension [1]. But how does the brain derive meaningful information from these

movements? Mouth movements provide information about phonological aspects of speech [2-3]. In contrast,

cospeech gestures display semantic information relevant to the intended message [4-6]. We show that when

63

language comprehension is accompanied by observable face movements, there is strong functional connectivity

between areas of cortex involved in motor planning and production and posterior areas thought to mediate

phonological aspects of speech perception. In contrast, language comprehension accompanied by cospeech gestures

is associated with tuning of and strong functional connectivity between motor planning and production areas and

anterior areas thought to mediate semantic aspects of language comprehension. These areas are not tuned to hand

and arm movements that are not meaningful. Results suggest that when gestures accompany speech, the motor

system works with language comprehension areas to determine the meaning of those gestures. Results also suggest

that the cortical networks underlying language

Vanduffel

Orban GA, Van Essen D, Vanduffel W. Comparative mapping of higher visual areas in monkeys and

humans. Trends Cogn Sci. 2004 Jul;8(7):315-24.

The advent of functional magnetic resonance imaging (fMRI) in non-human primates has facilitated comparison

of the neurobiology of cognitive functions in humans and macaque monkeys, the most intensively studied animal

model for higher brain functions. Most of these comparative studies have been performed in the visual system. The

early visual areas V1, V2 and V3, as well as the motion area MT are conserved in humans. Beyond these areas,

differences between human and monkey functional organization are increasingly evident. At the regional level, the

monkey inferotemporal and intraparietal complexes appear to be conserved in humans, but there are profound

functional differences in the intraparietal cortex suggesting that not all its constituent areas are homologous. In the

long term, fMRI offers opportunities to compare the functional anatomy of a variety of cognitive functions in the

two species.

Nelissen K, Borra E, Gerbella M, Rozzi S, Luppino G, Vanduffel W, Rizzolatti G, Orban GA. Action

observation circuits in the macaque monkey cortex. J Neurosci. 2011 Mar 9;31(10):3743-56.

In both monkeys and humans, the observation of actions performed by others activates cortical motor areas. An

unresolved question concerns the pathways through which motor areas receive visual information describing motor

acts. Using functional magnetic resonance imaging (fMRI), we mapped the macaque brain regions activated during

the observation of grasping actions, focusing on the superior temporal sulcus region (STS) and the posterior parietal

lobe. Monkeys viewed either videos with only the grasping hand visible or videos with the whole actor visible.

Observation of both types of grasping videos activated elongated regions in the depths of both lower and upper

banks of STS, as well as parietal areas PFG and anterior intraparietal (AIP). The correlation of fMRI data with

connectional data showed that visual action information, encoded in the STS, is forwarded to ventral premotor

cortex (F5) along two distinct functional routes. One route connects the upper bank of the STS with area PFG, which

projects, in turn, to the premotor area F5c. The other connects the anterior part of the lower bank of the STS with

premotor areas F5a/p via AIP. Whereas the first functional route emphasizes the agent and may relay visual

information to the parieto-frontal mirror circuit involved in understanding the agent's intentions, the second route

emphasizes the object of the action and may aid in understanding motor acts with respect to their immediate goal.

Vanduffel W, Fize D, Peuskens H, Denys K, Sunaert S, Todd JT, Orban GA. Extracting 3D from motion:

differences in human and monkey intraparietal cortex. Science. 2002 Oct 11;298(5592):413-5.

64

We compared three-dimensional structure-from-motion (3D-SFM) processing in awake monkeys and humans

using functional magnetic resonance imaging. Occipital and midlevel extrastriate visual areas showed similar

activation by 3D-SFM stimuli in both species. In contrast, intraparietal areas showed significant 3D-SFM activation

in humans but not in monkeys. This suggests that human intraparietal cortex contains visuospatial processing areas

that are not present in monkeys.

Wood

Action comprehension in non-human primates: motor simulation or inferential reasoning?

Wood & Hauser (2008) Trends in Cognitive Sciences Vol.12 No.12, 461-465.

Some argue that action comprehension is intimately connected with the observer’s own motor capacities,

whereas others argue that action comprehension depends on non-motor inferential mechanisms. We address this

debate by reviewing comparative studies that license four conclusions: monkeys and apes extract the meaning of an

action (i) by going beyond the surface properties of actions, attributing goals and intentions to the agent; (ii) by

using environmental information to infer when actions are rational; (iii) by making predictions about an agent’s

goal, and the most probable action to obtain the goal given environmental constraints; (iv) in situations in which

they are physiologically incapable of producing the actions. Motor theories are, thus, insufficient to account for

primate action comprehension in the absence of inferential mechanisms.

The Perception of Rational, Goal-Directed Action in Nonhuman Primates

Wood, Glynn, Phillips & Hauser Science 317, 1402 (2007)

Humans are capable of making inferences about other individuals' intentions and goals by evaluating their

actions in relation to the constraints imposed by the environment. This capacity enables humans to go beyond the

surface appearance of behavior to draw inferences about an individual's mental states. Presently unclear is whether

this capacity is uniquely human or is shared with other animals. We show that cotton-top tamarins, rhesus macaques,

and chimpanzees all make spontaneous inferences about a human experimenter's goal by attending to the

environmental constraints that guide rational action. These findings rule out simple associative accounts of action

perception and show that our capacity to infer rational, goal-directed action likely arose at least as far back as the

New World monkeys, some 40 million years ago.

The uniquely human capacity to throw evolved from a non-throwing primate: an evolutionary dissociation

between action and perception

Wood, Glynn & Hauser (2007) Biology Letters

Humans are uniquely endowed with the ability to engage in accurate, high-momentum throwing. Underlying this

ability is a unique morphological adaptation that enables the characteristic rotation of the arm and pelvis. What is

unknown is whether the psychological mechanisms that accompany the act of throwing are also uniquely human.

Here we explore this problem by asking whether free-ranging rhesus monkeys (Macaca mulatta), which lack both

the morphological and neural structures to throw, nonetheless recognize the functional properties of throwing.

Rhesus not only understand that human throwing represents a threat, but that some aspects of a throwing event are

more relevant than others; specifically, rhesus are sensitive to the kinematics, direction and speed of the rotating

arm, the direction of the thrower's eye gaze and the object thrown. These results suggest that the capacity to throw

did not coevolve with psychological mechanisms that accompany throwing; rather, this capacity may have built

65

upon pre-existing perceptual processes. These results are consistent with a growing body of work showing that non-

human animals often exhibit perceptual competencies that do not show up in their motor responses, suggesting

evolutionary dissociations between the systems of perception that provide understanding of the world and those that

mediate action on the world.

2011 Workshop on Action, Language and...

Documents

Transcript of 2011 Workshop on Action, Language and...