Post on 18-Apr-2018
2011 Workshop on
Action, Language and Neuroinformatics Hosted by the USC Brain Project, University of Southern California
July 25-27, 2011
Organizer: Michael A. Arbib
Database of Protocol-WrappedExperimental Results
Empirical Generalizations and Summary Tables and Figures
Models whichExplain and Predict
Pointers toAgreements
andDisagreements
Summarizationand Publication
Tools
Linksto RetrieveSupporting
Data
JournalArticles
Etc.
Venue: Hilton Checkers Hotel, 535 S. Grand Avenue | Los Angeles, CA 90071
This Workshop was supported in part by the National Science Foundation under Grant No. 0924674
(M.A. Arbib, Principal Investigator)
Website: http://neuroinformatics.usc.edu/mediawiki/index.php/Workshop
ii
Aims of the Workshop The Workshop is designed to address two linked research areas:
1) Neural mechanisms for the control, recognition and imitation of complex actions
2) Neurolinguistics, with an emphasis on embodied cognition and construction grammar informed by research on
the first topic
In doing this, we wish further to address:
3) Neuroinformatics focused on themes (1) & (2) by structuring
a) The key models from our groups and elsewhere.
b) Summaries of key data in a form which will aid the design and testing of models
with consideration of how best to develop new Collaboratory tools for the Brain Operation Database (BODB) and
federating them with other neuroinformatics resources.
We aim to discover new links between our different research programs, laying the basis for a number of new
collaborations. In addition to short presentations of research by all the participants, we will devote a full day to
structured discussions of the relations between Action, Language and Neuroinformatics. These discussions will lay
the basis for the publication of two reports:
1) Action Mechanisms in the Brain: Data, Models and Neuroinformatics
2) Language Mechanisms in the Brain: Data, Models and Neuroinformatics
Michael A. Arbib July, 2011
iii
Table of Contents Workshop Participants 1 Attendees’ Photos and Short Bios 2
Michael Arbib ...................................................................................................................................2 Lisa Aziz-Zadeh ................................................................................................................................2 Victor Barres .....................................................................................................................................3 Jason W. Bohland..............................................................................................................................3 James Bonaiuto .................................................................................................................................4 Bornkessel-Schlesewsky ...................................................................................................................4 Mihail Bota........................................................................................................................................5 Ramsay Brown ..................................................................................................................................5 Gully Burns .......................................................................................................................................6 Erica Cartmill ....................................................................................................................................6 Yiannis Demiris.................................................................................................................................6 Peter Fox ...........................................................................................................................................7 Brad Gasser .......................................................................................................................................8 Mike Grosvald...................................................................................................................................8 Madhuri Harway ...............................................................................................................................9 David Kemmerer ...............................................................................................................................9 Gerard Kempen ...............................................................................................................................10 Jinyong Lee .....................................................................................................................................11 Brian MacWhinney .........................................................................................................................11 David Marques ................................................................................................................................12 Risto Miikkulainen..........................................................................................................................12 Rick Misra .......................................................................................................................................13 Finn Årup Nielsen ...........................................................................................................................13 Erhan Oztop.....................................................................................................................................14 Matte Schilling ................................................................................................................................14 Rob Schuler .....................................................................................................................................15 Steve Small......................................................................................................................................15 Raymond Yu ...................................................................................................................................16 Wim Vanduffel................................................................................................................................16 Justin Wood.....................................................................................................................................17
Program 18 Day 1: Monday July 25 ...................................................................................................................18 Day 2: Tuesday, July 26 ..................................................................................................................18 Day 3: Wednesday July 27..............................................................................................................19
Abstracts of Presentations with Related Discussion Topics 20 Introduction .....................................................................................................................................20
Arbib: An Overview of the three themes .................................................................................20 Session 1..........................................................................................................................................20
Bonaiuto: Modeling Mirror Systems and More......................................................................20 Demiris: Modeling Imitation ....................................................................................................20 Kempen: Brain circuitry for syntax and syntactic processing in language production and
comprehension: Toward a neurocomputational model based on interactive activation and competition ......................................................................................................................21
Kemmerer: The cross-linguistic prevalence of SOV and SVO word order patterns reflects the hierarchical sequential representation of action in Broca's area.................................21
Schuler: New Developments in Collaboratory Workspaces for BODB ................................22 Session 2..........................................................................................................................................22
iv
Aziz-Zadeh: fMRI of the mirror system in action and language...........................................22 Small: Designing and Analyzing Naturalistic Biological Studies of Human Language.......22 Vanduffel: An integrated view of human and macaque data ................................................22 Bota: Revisiting the NeuroHomology Database......................................................................23 Marques: Wasn’t the web designed for data sharing? Our digital atlas and API program.23 Lee: Template Construction Grammar and the description of visual scenes.......................23
Session 3..........................................................................................................................................24 Bornkessel-Schlesewsky: Language diversity, actor-driven comprehension and
consequences for the neural language architecture.............................................................24 Barres: Viewing Template Construction Grammar as a Model of Comprehension ...........25 Grosvald: Dissociating linguistic and non-linguistic gesture processing: An ERP study of
American Sign Language.......................................................................................................26 Cartmill: From ape and human gesture to language..............................................................26 Schilling: Recruitment of Grounded Internal Models in Action, Cognition and Language26 MacWhinney: The Competition Model of language processing ............................................27 Burns: Building a Breakthrough Machine ..............................................................................28
Session 4..........................................................................................................................................28 Bohland: Informatics efforts to link gene expression data to speech/language brain systems28 Miikkulainen: Language and neural networks .......................................................................28 Oztop: From the concept of ‘self-observation for mirror neuron development’ to imitation
and robot skill transfer ..........................................................................................................29 Gasser: Ontogenetic Ritualization in nonhuman primates ....................................................29 Wood: How cognition functions in primates ...........................................................................30 Fox: BrainMap, ALE and meta-analytic modeling. ...............................................................30 Nielsen: The Brede database and new tools for meta-analysis ..............................................30
Background for the Workshop: Abstracts of Selected Papers by Participants 31 Arbib ...............................................................................................................................................31 Aziz-Zadeh ......................................................................................................................................32 Bohland ...........................................................................................................................................34 Bonaiuto ..........................................................................................................................................36 Bornkessel-Schlesewsky .................................................................................................................37 Bota .................................................................................................................................................39 Burns ...............................................................................................................................................40 Cartmill ...........................................................................................................................................41 Demiris ............................................................................................................................................43 Fox ..................................................................................................................................................44 Grosvald ..........................................................................................................................................46 Kemmerer........................................................................................................................................47 Kempen ...........................................................................................................................................49 Lee...................................................................................................................................................52 MacWhinney ...................................................................................................................................53 Miikkulainen ...................................................................................................................................55 Nielsen.............................................................................................................................................56 Oztop ...............................................................................................................................................58 Schilling ..........................................................................................................................................60 Small ...............................................................................................................................................61 Vanduffel.........................................................................................................................................63 Wood...............................................................................................................................................64
Workshop Participants
Overseas Participants
Ina Bornkessel-Schlesewsky <iboke@staff.uni-marburg.de> (Germany)
Yiannis Demiris <y.demiris@imperial.ac.uk> (England)
Gerard Kempen <gerard.kempen@mpi.nl> (Netherlands) Finn Aarup Nielsen <fn@imm.dtu.dk> (Denmark)
Erhan Oztop <erhan@atr.jp> (Japan)
Wim Vanduffel <Wim.Vanduffel@med.kuleuven.be> (Belgium)
North American Participants
Jason Bohland <jbohland@bu.edu> (Boston)
Erica Cartmill <cartmill@uchicago.edu> (Illinois)
Peter T Fox <fox@uthscsa.edu> (Texas)
David Kemmerer <kemmerer@purdue.edu> (Indiana)
Brian MacWhinney <macw@cmu.edu> (Pennsylvania)
David Marques <D.Marques@Elsevier.com> (San Francisco)
Risto Miikkulainen <risto@cs.utexas.edu> (Texas)
Rick Misra <R.Misra@elsevier.com>
Malte Schilling <mschilli@ICSI.Berkeley.EDU> (Berkeley)
Local Participants
Michael Arbib arbib@usc.edu
Lisa Aziz-Zadeh <lazizzad@usc.edu> (USC)
James Bonaiuto jimmy@vis.caltech.edu (Caltech)
Mihail Bota <mbota08@gmail.com> (USC)
Gully Burns <gully@usc.edu> (USC/ISI)
Mike Grosvald <mgrosval@uci.edu> (Irvine)
Steven L. Small small@uchicago.edu (Irvine)
Justin Wood <justin.wood@usc.edu> (USC)
Arbib’s Students
Jinyong Lee <jinyongl@usc.edu>
Robert Schuler <schuler@isi.edu>
Brad Gasser <bgasser@usc.edu>
Victor Barres <barres@usc.edu>
Madhuri Harway harway@usc.edu
Other Students
Ramsay Brown ramsaybr@usc.edu
Raymond Yu brendonfish@gmail.com (National University of Taiwan)
2
Attendees’ Photos and Short Bios
Michael Arbib
Michael A. Arbib is the Fletcher Jones Professor of Computer Science, as well as a Professor of Biological
Sciences, Biomedical Engineering, Electrical Engineering, Neuroscience and Psychology at USC. The thrust of his
work is expressed in the title of his first book, Brains, Machines and Mathematics (McGraw-Hill, 1964). The brain
is not a computer in the current technological sense, but he has based his career on the argument that we can learn
much about machines from studying brains, and much about brains from studying machines. His research has long
included a focus on brain mechanisms underlying the coordination of perception and action. His group prepared the
first computational model of mirror neurons and conducted some of the key initial imaging studies of the human
mirror system. He is now developing a new theory of the evolution of human language. He has also contributed to
neuroinformatics, with the current emphasis on BODB, the Brain Operation Database.
Lisa Aziz-Zadeh
Lisa Aziz-Zadeh's expertise is in cognitive neuroscience, with specific training in functional neuroimaging, the
mirror neuron system, and embodied semantics and cognition. As a graduate student at UCLA, she conducted
several fMRI and TMS studies on the mirror system, laterality, and language. Her training continued during her
post-doctoral fellowship at the University of Parma, Italy, where she utilized fMRI to study embodied semantics for
language as well as explored both covert and overt speech. She completed a second post-doc at UC Berkeley, where
together with Richard Ivry, Srini Naranayan and Jerome Feldman, she continued brain imaging work on embodied
semantics, supporting the previous finding to regions outside the motor cortex. In her current position at USC in the
Brain and Creativity Institute and the Division of Occupational Science and Occupational Therapy, Lisa has
3
continued exploring the neural basis of language processing using fMRI, in particular investigating how sensory
regions contribute to linguistic processing as well as other components of social cognition and emotion processing.
Victor Barres
Victor Barres is a PhD candidate in the neuroscience department at the University of Southern California under
the direction of Dr. Arbib. He received his M.S. degree in quantum physics from the Ecole Polytechnique, France in
2006, and his M.S degree in cognitive science from Ecole Normale Superieure, Paris in 2010. He focused his PhD
work on neurolinguistics and is currently working on a parsing model extending the SemRep/TCG system
developed by Lee and Arbib. His interests include computational models of cognitive systems, the role of motor and
spatial schemas in building semantic representations, and language emergence.
Jason W. Bohland
Jason W. Bohland is an Assistant Professor in the Department of Health Sciences at Boston University as well
as a faculty member in the cross-departmental Graduate Program for Neuroscience. I received my PhD in Cognitive
and Neural Systems from Boston University in 2007 with Dr. Frank Guenther, studying speech motor planning
using computational and neuroimaging techniques. I went on to a postdoctoral fellowship with Dr. Partha Mitra at
Cold Spring Harbor Laboratory, working generally in the area of neuroinformatics, with emphasis on understanding
large scale properties of the molecular and connectivity architecture of the brain, and helping to launch the Mouse
Brain Architecture Project, which aims to systematically map projection patterns in wild type and disease model
mice. At Boston University my laboratory brings together these distinct elements of my experience as we combine
computational, informatics, and neuroimaging methods to elucidate the structural and functional organization of
brain systems that support speech and language systems.
4
James Bonaiuto
James “Jimmy” Bonaiuto is a Swartz Postdoctoral Fellow at the California Institute of Technology. He received
a B.S. in Computer Science from Drexel University in 2004 and a Ph.D. in Neuroscience at the University of
Southern California in 2010. His doctoral thesis was completed under the advisement of Michael Arbib and was
entitled, “Modeling the Mirror System in Action Observation and Execution.” His research interests include
computational modeling of reaching and grasping, action observation, and imitation, synthetic brain imaging, and
neuroinformatics. Since 2010 he has been working as a postdoctoral researcher in Richard Andersen’s lab on fMRI
of spatial decision-making in awake, behaving monkeys.
Bornkessel-Schlesewsky
Ina Bornkessel-Schlesewsky is Professor of Neurolinguistics at the University of Marburg. She studied General
Linguistics and Computational Linguistics at the University of Potsdam and received her PhD from there in 2002.
From 2005 to 2010 she was the head of the research group "Neurotypology" at the Max Planck Institute for Human
Cognitive and Brain Sciences in Leipzig. In her research, she studies the neural bases of language comprehension,
with a particular focus on the implications of cross-linguistic diversity. To this end, she co-founded, together with
Matthias Schlesewsky, the new research field of neurotypology, which seeks to combine insights from
neurolinguistics and language typology in order to shed new light on the language architecture and how it is
implemented by the brain. A crucial part of this work has lain in the development of a cross-linguistically adequate
neurolinguistic model of language comprehension, the extended Argument Dependency Model (eADM). Current
research interests including broadening the scope of the eADM's coverage to include the discourse level and
grammatical relations, predictions for neuroanatomy and the use of naturalistic stimuli.
5
Mihail Bota
Mihail Bota is an Associate (Research) Professor in the Department of Neurobiology, University of Southern
California. His former education include Physics (Biophysics) and Philosophy. His Ph.D. Thesis was a study of the
neural homologies of the rat, monkey and human cortex, with application to the premotor cortex . He was the
developer and main collator and curator of the NeuroHomology Database, under the supervision of Professor
Michael Arbib. His more recent work includes development and population of the Brain Architecture Management
System (BAMS; http://brancusi.usc.edu/bkms; http://brancusi1.usc.edu), which is an online publicly accessible
neuroinformatic workbench that handles data across several levels of the mammalian central nervous system. His
research interests include construction of functionally relevant neural circuits from data collated from the literature,
connectome construction and comparison of neural features across different mammalian species, and construction of
controlled vocabularies (ontologies) as standards for interoperability across different neuroinformatic platforms.
Ramsay Brown
Ramsay Brown is a graduate student at the University of Southern California. He earned his bachelors at USC in
Neuroscience, and has been admitted to the Neurobiology PhD Program starting this fall. Working with Dr. Larry
Swanson and Dr. Mihail Bota, Ramsay is actively developing the "flatmap" online graphical user interface to the
Brain Architecture Management System and looks forward to its launch this coming year. He is interested in how
neural networks found in the brain underlie goal-oriented survival behavior, the exploration of network structure
through experimental neuroanatomy, and what network topology can tell us about network function.
6
Gully Burns
Gully received an undergraduate degree in Physics from Imperial College in London, and a Doctoral degree in
Physiology from Oxford University. He is interested in developing Knowledge Engineering methods for Bio-
medicine that allow scientists to reason and process more knowledge. His main research interests are to understand
and develop systems that enable and even encourage scientists to make breakthroughs.
Erica Cartmill
Erica Cartmill is a postdoctoral scholar in the Department of Psychology at the University of Chicago. She works
with Professor Susan Goldin-Meadow on the role that gesture plays in language acquisition and in grounding speech
in the physical world. Erica earned her PhD in 2009 from the University of St Andrews under the supervision of
Professor Richard Byrne for her research on orangutan gestural communication. Her work with both children and
apes aims to understand environmental influences on communication/language and the role gesture plays in the
development of communication systems on both evolutionary and ontogenetic timescales.
Yiannis Demiris
7
Yiannis Demiris heads the Bioinspired Assistive Robots and Teams (BioART) laboratory at the Department of
Electrical and Electronic Engineering of Imperial College London, doing research on human-robot interaction,
assistive robotics, and multirobot teams, with an emphasis on biologically inspired mechanisms of social
development and social learning. He received his PhD in Intelligent Robotics from the Department of Artificial
Intelligence of the University of Edinburgh in 1999, and joined the faculty of Imperial College in 2001. His current
projects include two European FP7 projects, ALIZ-E (Adaptive Strategies for Sustainable Long-Term interaction)
and EFAA (Experimental Functional Android Assistant) where he is aiming to develop bio-inspired mirror and
imitation systems for enabling robots to perceive, learn from and assist human actions in multimodal tasks, ranging
from classical assembly tasks to musical instrument playing and dance with adult and children subjects.
Peter Fox
Dr. Fox is the Director of the Research Imaging Institute (RII; http://rii.uthscsa.edu/) of the University of Texas
Health Science Center at San Antonio (UTHSCSA). His primary academic appointment is in the Department of
Radiology, in which he holds the Malcolm Jones Chair and serves as the Vice Chair of Research and Research
Education. He holds joint appointments in Neurology, Psychiatry and Physiology and holds research appointments
at the South Texas Veterans Health Care System and the Southwest National Primate Center. Hehas been engaged in
neuroimaging research for over 30 years, starting his career at the Radiological Sciences Division of the
Mallinckrodt Institute at Washington University, St. Louis. Dr. Fox pioneered the use of standardized coordinates
for reporting neuroimaging results and created the BrainMap database (www.brainmap.org), which is a resource for
meta-analysis and modeling of functional and structural neuroimaging results. He received the UTHSCSA
Presidential Research Scholar Award, is a Fellow of the AAAS and has been recognized by ISI as one of the most
highly cited neuroscientists. Current interests include: modeling neural systems from neuroimaging data; meta-
analysis methods; optimal strategies for assessing treatment effects in neurological and psychiatric disorders;
hemodynamic and metabolic physiology and pathophysiology.
8
Brad Gasser
Brad Gasser is a Ph.D. student in the Neuroscience Graduate Program at the University of Southern California. I
am working with Dr Michael Arbib on building computational models of primate brain systems involved in
behavioral control and social learning. In particular, we are concerned with the functional brain changes in the
hominid line that would lead to the 'language-ready brain' in homo sapiens. I received degrees in philosophy and
psychology at Bowling Green State University in Ohio, where I worked with Dr Verner Bingman recording
hippocampal neurons in pigeons. I was also an NSF-sponsored intern with Dr David Yager at the University of
Maryland, College Park studying praying mantid escape behavior in flight.
Mike Grosvald
Mike Grosvald completed his MA in Mathematics at UC Berkeley and his PhD in Linguistics at UC Davis; his
dissertation was a production and perception study of coarticulation in spoken and signed language. As a PhD
student working in David Corina's lab, he also worked on a number of other projects, using both behavioral and ERP
methodology, investigating how signs and other human actions are processed by signing and sign-naive individuals.
In 2010, he began working in the Solodkin/Small lab, where he is using behavioral and fMRI methodology to
explore issues related to audio-visual integration (at the word level), as well as semantic and syntactic processing (at
the sentence level).
9
Madhuri Harway
Madhuri Harway is doing her Master's degree in Electrical Engineering at USC. Inspired by Richard Byrne's
work on imitation by behavioral parsing in young gorillas, she has made a set of videos of people making tea in the
Indian style, and is annotating these as the basis for assessing learning methods that could perform behavioral
parsing on widely different performances of the same skill.
David Kemmerer
I have a joint appointment in the Department of Speech, Language, and Hearing Sciences and the Department of
Psychological Sciences at Purdue University. My teaching responsibilities include courses on the neural bases of
speech and language, the field of cognitive neuroscience, and topics in linguistics. My research focuses on how
different kinds of linguistic meaning are mediated by different neural systems, drawing on behavioral and lesion
data from brain-damaged patients as well as behavioral and functional neuroimaging data from healthy subjects. My
current projects include the linguistic encoding of action and the syntax-semantics interface. I also like to follow
literature on human evolution and on various non-linguistic topics in cognitive neuroscience, such as the neural
correlates of consciousness. Mostly, though, I've been consumed with writing a textbook called "The cognitive
neuroscience of language: An introduction," under contract with Psychology Press. My hobbies include road biking,
cooking, exploring the back roads of Indiana, and reading novels, short stories, and "popular science" books.
10
Gerard Kempen
Gerard A.M. Kempen (1943) has been professor of cognitive psychology at Leiden University since 1992
(emeritus since 2008), and Research Associate of the Max Planck Institute for Psycholinguistics in Nijmegen since
1999. From 1976 to 1992 he was professor of psycholinguistics at the Radboud University in Nijmegen, where he
had received his PhD in 1970.
His psycholinguistic work concerns the grammatical aspects of human sentence processing during language
production and comprehension. He is studying these processes through a combination of experimental-psychological,
linguistic, and computational methods. His contributions to this topic include:
• the division of labor and the interaction between conceptualization and formulation processes in sentence
production (1977)
• the concept of incrementality in language production (1982)
• the distinction between lemmas and lexemes in the Mental Lexicon for language production (1983)
• an early computational model of human sentence production (1987)
• the (neuro)cognitive model of parsing called Unification Space (1989-2009)
• the (neuro)cognitively motivated Performance Grammar formalism (1991-2003)
• a grammatical theory of clausal coordinate ellipsis based on similarities with speech error repairs (2009)
• experimental and theoretical arguments for large-scale overlap between the grammatical processing
mechanisms underlying sentence comprehension and those underlying sentence production (2000-2011)
• a new model of self-monitoring for grammatical and lexical speech errors, which does not require the
“perceptual loop” (2011).
Since 1980 he initiated and supervised various theoretical and applied research projects dealing with the
computational treatment of Dutch, among other things, for visual-interactive teaching of grammatical structures
(sentence analysis) in secondary education.
11
Jinyong Lee
Jinyong Lee is a Ph.D. student in the Department of Computer Science of the University of Southern California.
He received his B.S. degree in Electrical and Computer Engineering in 2005 from Hanyang University in Seoul,
Korea and the M.S. degree in Computer Science from the University of Southern California in 2007. His research
interests include modeling human language production and comprehension, semantic representation,
neurolinguistics and psycholinguistics. He is currently working on a model of production of scene description. This
work involves building a computational model (Template Construction Grammar and its extension) with a new type
of semantic representation, SemRep, as well as conducting a set of eye-tracking experiments to collect supporting
evidence for the model.
Brian MacWhinney
Brian MacWhinney is Professor of Psychology, Computational Linguistics, and Modern Languages at Carnegie
Mellon University. He received a Ph.D. in psycholinguistics in 1974 from the University of California at Berkeley.
He has developed a model of first and second language processing and acquisition based on competition between
item-based patterns. In 1984, he and Catherine Snow co-founded the CHILDES (Child Language Data Exchange
System) Project for the computational study of child language transcript data. The CHILDES programs and database
have now become an important component of the basic methodology of research in language acquisition. He is now
extending this system to six additional research areas in the form of the TalkBank Project. MacWhinney’s recent
work includes studies of online learning of second language vocabulary and grammar, neural network modeling of
lexical development, fMRI studies of children with focal brain lesions, and ERP studies of between-language
competition. He is also exploring the role of grammatical constructions in the marking of perspective shifting, the
determination of linguistic forms across contrasting time frames, and the construction of mental models in scientific
reasoning.
12
David Marques
David Marques is currently VP of Business Development for the Books group in the Science and Technology
division of Elsevier. He has been at Elsevier since 1999, when he took the job of Chief Technology Officer and
created and managed Elsevier Labs and Elsevier User-Centered Design. David holds an A.B. degree in Psychology
from Cornell University (1972), a Ph.D. in Psychobiology from Michigan (1976), and did post-doctoral research in
neuroscience at Pittsburgh and at the Worcester Foundation for Experimental Biology. In 1981, David took a left
turn into technology, working for 15 years at Digital Equipment Corporation in software consulting, Artificial
Intelligence Research, Education market technology, and collaboration software at AltaVista. His most recent work
has been to lead the product development of Elsevier’s BrainNavigator.
Risto Miikkulainen
Risto Miikkulainen is a Professor of Computer Sciences at the University of Texas at Austin. He received an
M.S. in Engineering from the Helsinki University of Technology, Finland, in 1986, and a Ph.D. in Computer
Science from UCLA in 1990. His current research includes models of natural language processing, self-organization
of the visual cortex, and evolving neural networks with genetic algorithms; he is an author of over 250 articles in
these research areas. He is currently on the Board of Governors of the Neural Network Society, and an action editor
of IEEE Transactions on Autonomous Mental Development, IEEE Transactions on Computational Intelligence and
AI in Games, the Machine Learning Journal, Journal of Cognitive Systems Research, and Neural Networks.
13
Rick Misra
Kaushik (Rick) Misra is a member of Elsevier’s BrainNavigator product development team and a collaborating
scientist at the Scripps Research Institute. His Ph.D. involved investigating the molecular and genetic mechanisms
of alcohol addiction and mood disorders. His first post-doctoral work used siRNA, antisense, and pharmacological
blockade techniques to examine the interaction between neuropeptide and neurotrophic systems. He later took a
post-doctoral position at the Scripps Research Institute using behavioral, molecular, and pharmacological techniques
as a means of examining the involvement of neuropeptides associated with pain and stress systems in opiate
dependence. He began work in the field of neuroinformatics and neuroscience based analytic tools as a product
manager for Elsevier’s BrainNavigator. His particular focus remains creating an interface that utilizes neuroanatomy
as a platform for structure based neuroinformatics.
Finn Årup Nielsen
Finn Årup Nielsen is Senior Researcher at the Technical University of Denmark, where he also completed his
PhD. His dissertation was entitled "Neuroinformatics in Functional Neuroimaging" and he has since been working
on combining scientific and web information both in the form of numerical data, text and ontology components. He
has developed two Matlab-based toolboxes: One ("Lyngby") for primarily analyzing original neuroimaging data in a
variety of ways, another ("Brede") for alternative analysis with text mining and meta-analysis and handling of Web
data. Initially working with the BrainMap database, he developed the Brede Database and now focuses on a wiki-
oriented approach with the Brede Wiki for storing, quering and analyzing numerical data from scientific studies.
14
Erhan Oztop
Erhan Oztop earned his Ph.D. at the University of Southern California in 2002. In the same year, he joined the
Computational Neuroscience Laboratories at Advanced Telecommunications Research Institute International (ATR)
in Japan where he worked as a researcher until 2004. Immediately after, he joined the JST ICORP Computational
Project as a researcher, and later became a group leader (2004-2008). Currently he works at NICT, Advanced ICT
Research Institute, Brain ICT Laboratory and is a senior researcher of ATR Cognitive Mechanisms Laboratories
where he is leading the Communication and Cognitive Cybernetics group. He also holds a visiting associate
professor position at Osaka University. His research interests include computational and cognitive neuroscience,
human-robot interaction and computational modeling and analysis of intelligent behavior.
Matte Schilling
Malte Schilling is a PostDoc at the International Computer Science Institute (ICSI) in Berkeley. His work
concentrates on internal models, their grounding in behaviour and their application in higher-level cognitive function
like planning ahead or communication.
Before moving to Berkeley, he worked at the Sony Computer Science Laboratory in Paris on the embodiment of
language and the connections between internal models and language.
He received his PhD in Biology at the University of Bielefeld in January, 2010. His PhD project focuses on the
control of a hexapod robot, applying insights from biological experiments with stick insects on a reactive control
model, and enhancing this control through cognitive capabilities by introducing an internal body model implemented
as a neuronal network. After finishing his PhD he became a Responsible Investigator at the Center of Excellence
'Cognitive Interaction Technology' (CITEC), University of Bielefeld.
15
He has studied Computer Science (artificial intelligence as a main subject) from 1997-2003 at the University of
Bielefeld and finished the Diploma with his thesis on knowledgebased systems in the context of virtual
environments.
Rob Schuler
Rob Schuler is a Ph.D. candidate in the Dept. of Computer Science of the University of Southern California. He
received the B.S. degree in Computer Science in 1995 and the M.S. degree in Computer Science in 2008, both from
USC. His research interests include scientific data management, scientific collaboratories, and databases and tools to
support neuroscience. Since 2004, Rob has worked as a systems programmer at the USC Information Sciences
Institute working on large-scale data management systems for biomedical, physical, and climate sciences. Most
recently, he has served as a member of the steering committee of the Biomedical Informatics Research Network
(BIRN). Previously, he held a variety of positions in the technology industry including trusted systems development
for a division of Xerox Research & Technology.
Steve Small
Steven L. Small is the Stanley van den Noort Professor and Chair of Neurology, as well as Professor of
Neurobiology and Behavior and Cognitive Sciences at the University of California, Irvine. He is also Professor
Emeritus of Neurology and Psychology at The University of Chicago, and serves as Editor in Chief of the
international journal Brain and Language. The main focus of his efforts over the past two decades has been in the
neurobiology of language, and he has recently founded an international organization dedicated to this topic, the
Society for the Neurobiology of Language, and an associated conference, the Neurobiology of Language Conference,
which recently had its second annual meeting with over 400 participants. Why a new society and conference? The
emphasis of work from this new perspective is to uncover the neural mechanisms of language, i.e., the specific
implementations of language functions in neurons, axons, glial cells, electrical signals, chemical messengers, genetic
16
expression, and related biological entities. This differs from the other approaches to the study of language that use
brain measurements, in which the focus is on cognitive systems, linguistic theories, and other non-biological
explanations. In Small's laboratory, research has primarily used functional MRI and has investigated mechanisms of
speech perception and naturalistic language comprehension, with a recent emphasis on the extent to which sensory
and motor neural codes are involved in everyday language use.
Raymond Yu
Raymond Yu is a Ph.D. candidate in the Department of Computer Science, Nation Taiwan University, Taipei,
Taiwan. His research interests are in constructing computational models which help to understand the function of the
brain, especially functions related to the human visual process. Currently he is working on a computational model
involving prefronto-parietal interactions for the cognitive functions of visual-spatial sketchpad, counting, and scene
integration. This work was initiated during the calendar year 2010 when he was a Visitor in Michael Arbib's lab at
USC.
Wim Vanduffel
Wim Vanduffel conducts research at the Lab for Neuro-and Psychophysiology, K.U.Leuven, Belgium, as well as
being Assistant in Neuroscience at MGH and Assistant Professor in Radiology, Harvard Medical School. He has
extensive expertise in the use of neuroimaging to study humans and awake behaving monkeys. His specific area of
interest is the primate visual system. He compares directly the functional organization of the visual system in human
and non-human primates using the fMRI technique and novel comparative methods. He currently focuses on the
functional role of feedback connections within the attention system by performing ‘perturb-and-measurement’
techniques in non-human primates. In particular, he investigates the behavioral and functional consequences of
17
reversible (in)activations of a node within a cortical network. He also studies the effect of attention and reward
signals on sensory processing.
Justin Wood
Assistant Professor of Psychology, College of Letters, Arts, & Sciences. Educational Background: University of
Virginia, Harvard University. Research examines the origins of the mind in relation to how knowledge emerges
during development and may have emerged on a biological scale. Studies a wide range of cognitive abilities,
including mathematics, object perception and cognition, and the social systems that allow humans and nonhuman
animals to make inferences about others' goals, intentions, beliefs, and desires Recipient of New Investigator Award
from the American Psychological Association
18
Program
All sessions will be held in Room A except that breakout sessions will have their separate rooms on Wednesday.
Coffee will be available from 8:30 in Room A on Monday and Tuesday and in the breakout rooms on
Wednesday.
The banquet will be held on the Checkers Patio, with a Reception at 7 and seating for dinner at 7:30.
Day 1: Monday July 25
9:00-10:00: Arbib – An Overview of the three themes
10:00-1:00: Session 1: Five 30 minute talks: [Coffee break 10:30-11:00]
Bonaiuto: Modeling Mirror Systems and More
Demiris: Modeling Imitation
Kempen: Brain circuitry for syntax and syntactic processing in language production and comprehension:
Toward a neurocomputational model based on interactive activation and competition
Kemmerer: The cross-linguistic prevalence of SOV and SVO word order patterns reflects the hierarchical
sequential representation of action in Broca's area
Schuler: New Developments in Collaboratory Workspaces for BODB
1:00-2:00: Lunch
2:00-5:30: Session 2: Six 30 minute talks [Coffee break 3:30-4:00]
Aziz-Zadeh: fMRI of the mirror system in action and language
Schilling: Recruitment of Grounded Internal Models in Action, Cognition and Language
Vanduffel: An integrated view of human and macaque data
Bota: Revisiting the NeuroHomology Database
Marques: Wasn’t the web designed for data sharing? Our digital atlas and API program
Lee: Template Construction Grammar and the description of visual scenes
Day 2: Tuesday, July 26
9:00-1:00: Session 3: Seven 30 minute talks [Coffee break 10:30-11:00]
Bornkessel-Schlesewsky: Language diversity, actor-driven comprehension and consequences for the
neural language architecture
Barres: Viewing Template Construction Grammar as a Model of Comprehension
Grosvald: Dissociating linguistic and non-linguistic gesture processing: An ERP study of American Sign
Language
Cartmill: From ape and human gesture to language
Small: Designing and Analyzing Naturalistic Biological Studies of Human Language
MacWhinney: The Competition Model of language processing
Burns: Building a Breakthrough Machine
1:00-2:00: Lunch
2:00-6:00: Session 4: Seven 30 minute talks: [Coffee break 3:30-4:00]
Bohland: Informatics efforts to link gene expression data to speech/language brain systems
19
Miikkulainen: Language and neural networks
Oztop: From the concept of ‘self-observation for mirror neuron development’ to imitation and robot skill
transfer
Gasser: Ontogenetic Ritualization in nonhuman primates Wood: How cognition functions in primates
Fox: BrainMap, ALE and meta-analytic modeling
Nielsen: The Brede database and new tools for meta-analysis
6:00-7:00 Free time
7:00Reception
7:30: Banquet
Day 3: Wednesday July 27
9:00-11:30: Three parallel Breakout Sessions 1: Cross-cutting groups. [Refreshments in meeting rooms.]
[The name of the rapporteur for each session is marked in Bold.]
1a. Half the Action group + half the Neuroinformatics group: assessing the model-data integration and
neuroinformatics needs of the Action group: Bonaiuto, Oztop, Demiris, Vanduffel, Arbib, Marques, Bohland, Bota
1b. Half the Neurolinguistics group + the other half of the Neuroinformatics group: assessing the model-data
integration and neuroinformatics needs of the Neurolinguistics group: Bornkessel-Schlesewsky, Small,
MacWhinney, Miikkulainen, Nielsen, Fox, Barres, Schuler
1c. The other half of the Action group + the other half of the Neurolinguistics group: Defining shared modeling
challenges and the development of a shared conceptual framework. Kemmerer, Aziz-Zadeh, Cartmill, Gasser,
Grosvald, Wood, Kempen, Lee, Schilling.
11:30-12:30: Lunch
12:30-2:30: Three parallel Breakout Sessions 2: One group for each theme meet separately.
2a. Action group: What are the key data and/or conceptual issues ripe for modeling; what are the key lines of
modeling that hold most promise to address these data/issues? Oztop, Demiris, Vanduffel, Cartmill, Arbib, Aziz-
Zadeh, Gasser, Schilling, Wood
2b. Neurolinguistics group: What are the key data and/or conceptual issues ripe for modeling; what are the key
lines of modeling that hold most promise to address these data/issues? MacWhinney, Kempen, Grosvald, Small,
Bornkessel-Schlesewsky, Miikkulainen. Kemmerer, Lee, Barres
2c. Neuroinformatics group: What tools are ripe for sharing, or should be ripened? What are promising lines for
federation? Nielsen, Marques, Bohland, Bonaiuto, Bota, Fox, Schuler
2:30-3:00: Coffee break.
3:00-5:00: Concluding Plenary Session: Each rapporteur will present a 10 minute report on the findings of his/her
group followed by 10 minutes of general discussion. These reports will lay the grounds for future research and
collaboration.
5:00 Adjourn
20
Abstracts of Presentations with Related Discussion Topics
Introduction
Arbib: An Overview of the three themes This will provide a brief introduction to our work on modeling neural mechanisms for action and action
recognition; an evolutionary scenario linking action, action recognition, imitation and language; and our Brain
Operation Database, BODB. Synthetic Brain Imaging (SBI) will be introduced as a way to integrate modeling based
on data from the macaque and human brain. The presentation will provide a framework for the following
presentations and discussion sessions.
Session 1
Bonaiuto: Modeling Mirror Systems and More Both premotor and parietal cortex of the macaque brain contain mirror neurons each of which fires vigorously
both when the monkey executes a certain limited set of actions and when the monkey observes some other perform a
similar action. Turning to the human, we must rely on brain imaging rather than single-neuron recording. I will
present biologically plausible models of the mirror system and its interactions with other brain regions in grasp
observation and execution. The Mirror Neuron System (MNS2) model suggests an extension to the original model
that addresses recent data on the response of mirror neurons to sounds characteristically associated with an action,
and to grasps in which the final portion is occluded by an obstruction. The Integrated Learning of Grasps and
Affordances (ILGA) model gives an account of how F5 canonical and AIP visual neurons, which provide input to
the MNS2 model, simultaneously gain their properties through development. Augmented Competitive Queuing
(ACQ) then shows how systems like ILGA and MNS2 can be deployed and used to chain together sequences of
actions and rapidly reorganize these sequences in the face of disruption. Finally I will briefly discuss the use of
synthetic brain imaging to allow computational models to address monkey and human data.
Additional Discussion Points for Wednesday
Extending the above models to models of imitation.
Further development of SBI, and links to work of Aziz-Zadeh and Vanduffel .
Challenges for BODB development.
Demiris: Modeling Imitation For the last few years, I have been working on modeling mechanisms of social learning and imitation trying to
simultaneously stay close to biological data as well as implementing the mechanisms on robotic systems. Starting
from data on imitation in adults and infants, data on visuo-imitative apraxia, as well as gaze tracking during human
action perception, I proposed a model that involves a dual-route process for imitation and understanding actions,
involving both passive (learning) and active (mirror-like) computational elements. Instantiation of the model over
several years on a variety of tasks (on robots, adversarial computer games and assistive robotic wheelchairs)
revealed a number of lessons for modellers including the need for perspective taking, hierarchical processing, and
principled control of attention during action perception, and put forward a number of testable predictions regarding
the behavior of mirror neurons.
21
Additional Discussion Points for Wednesday
Linking to Oztop and Gasser: How can this model inform computational neuroscience?
Role of working memory and attention
Positive transfer to structurally similar tasks: cf. Wood & Cartmill
Perspective taking
Computational strategies for social learning and imitation; hierarchical representations for behavioral control and
strategies for learning them.
Kempen: Brain circuitry for syntax and syntactic processing in language production and
comprehension: Toward a neurocomputational model based on interactive activation and
competition Space model of grammatical decoding (parsing). Presentation of new behavioral and fMRI evidence in favor of
the idea that grammatical decoding in sentence comprehension utilizes the same neurocognitive infrastructure as
grammatical encoding in sentence production. Discussion of implications for the design of a direction-neutral
neuronal mechanism for grammatical decoding *and* encoding.
Additional Discussion Points for Wednesday
Compare/contrast Performance Grammar and Construction Grammar.
Could direction-neutral grammatical encoding/decoding involve mirror neurons?
Plausibility of self-organizing or optimization mechanisms other than interactive activation and competition as
models of syntactic processing.
Response to Bornkessel-Schlesewsky critique of 2006.
Relation to MacWhinney competition model.
Kemmerer: The cross-linguistic prevalence of SOV and SVO word order patterns reflects the
hierarchical sequential representation of action in Broca's area It is well-established that the vast majority of the roughly 7,000 languages in the world have either SOV (about
48%) or SVO (about 41%) word order (Dryer, 2011). Several scholars have argued that these strong tendencies can
be explained cognitively in terms of the prototypical transitive action scenario, in which an animate agent acts
forcefully on an inanimate patient and thereby induces a change of state. Two forms of iconicity are especially
relevant: first, because the agent is at the head of the causal chain that affects the patient, subjects usually precede
objects; and second, because it is the agent’s action, rather than the agent per se, that changes the state of the patient,
verbs and objects are usually adjacent. My aim is to show that this account can be deepened and enriched by relating
it to recent research on how actions are represented in the brain. Specifically, I review several lines of evidence
which, taken together, suggest that Broca's area—more precisely, BA44—plays a pivotal role in representing the
hierarchical sequential organization of goal-oriented bodily movements, not only when they are performed and
perceived, but also when they are linguistically described. Based on these findings, I propose that the most cross-
linguistically prevalent word order patterns reflect the most natural ways of selecting and nesting action components
in BA44.
Additional Discussion Points for Wednesday
22
Relations to Aziz-Zadeh and Vanduffel on the MNS.
Relation to Bornkessel-Schlesewsky's work on word order
Relation to MacWhinney's work on starting points and perspectives
Relation to Lee’s linkage of vision and language, and the implication of different styles of action and related
verbs.
Schuler: New Developments in Collaboratory Workspaces for BODB The talk will assume familiarity with basic design ideas of BODB (e.g., BOPs, SEDs, SSRs, and present current
work on developing Workspace resources to promote collaborations between modelers and experimentalists on
specific topics in developing summaries of related empirical data and building models which address them.
Additional Discussion Points for Wednesday
What Workspaces could be developed for different subsets of participants at the Workshop to foster future
collaboration?
How might BODB best be federated with the other databases discussed at the Workshop so that their data can be
deployed to help efforts in modeling?
Session 2
Aziz-Zadeh: fMRI of the mirror system in action and language The Human Mirror Neuron System, Embodied Representation & Embodied Semantics; relation to the
'mentalizing' systems in humans
Additional Discussion Points for Wednesday
Relating human data to insights from monkey studies.
Relation of MNS and mentalizing systems to (medial) cognitive control networks
What repair and rehabilitation can tell us about large scale brain networks
Challenges of linking relevant data sets from fMRI, TMS, and neuroanatomy.
Small: Designing and Analyzing Naturalistic Biological Studies of Human Language Addressing the use of fMRI in investigations of the neurobiology of language. This requires a substantial
reworking of design and analysis methods for fMRI, and modeling the peaks and valleys in raw fMRI time series is
a valuable approach.
Additional Discussion Points for Wednesday The relation between action understanding and language comprehension.
The relation of fMRI and ERP data.
Vanduffel: An integrated view of human and macaque data Recent models predicted that human-specific cognitive abilities emerged from recycled neuronal circuits. Instead,
evolutionary models based on cortical expansion suggested that human-specific functions are carried by novel
associative regions, without correspondent areas in other species. Yet no direct experimental evidence exists to
support either hypothesis. Using novel comparative tools which I will discuss in detail, we revealed spatially
correspondent brain networks, such as the language network, with different functional processing as predicted by the
recycling theory. In regions with larger cortical expansion, however, we observed human-specific networks with
23
completely distinct functional signatures compared to the macaque. Importantly, their presumed functions suggest a
direct link with human-specific cognitive abilities, such as abstract reasoning. Thus both neuronal recycling and
cortical expansion account for the emergence human-specific cognitive abilities.
Additional Discussion Points for Wednesday
Towards model-free methods for comparative functional studies
How to decide that a particular cognitive function is human-specific?
Are 'human-specific' cognitive functions carried by novel brain regions or by recycled brain regions?
Issues specific and common to neuroinformatics with monkey and human imaging data
Link to Aziz-Zadeh, relating MNS to other regions; Bota on neo NHDB
Bota: Revisiting the NeuroHomology Database This talk will be composed of three parts. First, I will review the NeuroHomology Database, its scope, addressed
challenges, and the results obtained by data mining. The second part will be a review of the latest development of
BAMS. Finally, I will address the possible points of convergence of these two systems, towards creation of a new
neuroinformatic system, neoNHDB.
Additional Discussion Points for Wednesday
Vanduffel as a customer of neoNHDB
Challenges of collating data for action, action recognition in macaque and human and relating them to
neurolinguistic data.
Progress and possibilities for federating BAMS with other resources.
Marques: Wasn’t the web designed for data sharing? Our digital atlas and API program. In the past 2 years, we have put 4 stereotaxic atlases online, organized by a new ontogeny-defined brain structure
ontology (from Charles Watson, Luis Puelles, George Paxinos) and toolsets to allow comparison with researcher
histology and to help visualize structures and fiber tracts in 3D to facilitate intervention planning and interpretation
accuracy. Toward our vision of helping bring together structure and function information to enhance understanding,
we link out to structure-specific data sources such as BODB and BAMS, provide APIs for others to retrieve
structure and atlas information, and are building public MRI templates to help co-register subjects to the reference
atlases. This paper presents details of how the models were constructed and the APIs that are available, and poses
the question: What APIs would be of value, where should this vision go? Additional Discussion Points for Wednesday
Work with Bonaiuto on the Macaque Brain Device.
Linking across different atlases and species
Suggestions for encouraging adoption of BODB
Link via Allen Brain Atlas to Bohland.
Lee: Template Construction Grammar and the description of visual scenes A computational model of production of descriptions of visual scenes was previously outlined by Arbib & Lee
(2008). One part of this effort was the development of a new kind of semantic representation, SemRep. SemRep is
an abstract form of visual information with an emphasis on the spatial linkage of entities, attributes and actions.
SemRep provides a graph-like hierarchical structure with enough formal semantics for verbal description of a scene.
24
Moreover, Construction Grammar was adopted as an appropriate framework for a schema-based linguistics, and
Template Construction Grammar (TCG) has been implemented as an operational version of such a framework.
Constructions, represented as schema instances, compete and cooperate to cover the SemRep to produce a
description of the visual scene at hand. The present study reports on a language model which expands the previous
account of SemRep and TCG by providing a particular mechanism of working memory (WM) and attentional focus.
The proposed mechanism, which is closely related to Cowan’s (1999) WM model, blends the focus of attention with
the WM capacity by having what is being attended and what is stored in the WM identical. In this approach, the
vision system interprets a part of the scene under the current focus of attention, and then the SemRep is created or
updated accordingly in the WM. The language system applies constructions on the SemRep built in the WM
according to the TCG rule. Therefore, the deployment mechanism of attentional focus is hypothesized to be the main
driving force of the resultant sentential structure being produced. The notion of subscene, whose preliminary
definition was provided previously through the account of the minimal subscene (Itti & Arbib, 2006), is proposed as
the unit of attentional process. A subscene is an area covered by entities of the scene under the current attentional
focus, which comes with various spanning coverage from parts of an entity to the entire scene. SemReps
corresponding to subscenes with different attentional spans are hierarchically organized and processed by the system
as attentional focus plays a role as an executive window while travelling across the hierarchy by zooming in/out and
shifting. The WM capacity limits the number (and quality) of entities that are simultaneously captured by the
attentional focus and processed in WM while the structure of the produced sentences is affected by a threshold value
for readout of constructions being assembled in WM – a lower threshold value would result in fragmented phrases
whereas a higher value would yield more complete sentences. Also, preliminary evidence from an eye-tracking
experiment is discussed.
Additional Discussion Points for Wednesday
Vision: Links to Vanduffel and the challenge of finding neural correlates. Relation to Kemmerer.
Eye movements and EyeParser. Link to neuroinformatics?
Compare/contrast Embodied Construction Grammar (Schilling) and Fluid Construction Grammar
Session 3
Bornkessel-Schlesewsky: Language diversity, actor-driven comprehension and consequences
for the neural language architecture With over 6000 languages spoken in the world today, one of the most striking characteristics of human language is
surely the diversity of its manifestations. In view of this high variability, it is perhaps not surprising that absolute
linguistic universals are virtually non-existent. However, linguistic variation is also not arbitrary, with some patterns
occurring more frequently than others. These skewings towards certain patterns are likely tied to the way in which
language is implemented in the brain.
Building on this working hypothesis, I will present salient aspects of the latest version of a neurolinguistic
model of language processing, the extended Argument Dependency Model (eADM). The eADM follows in the
footsteps of MacWhinney and Bates' Competition Model in assuming that fine-grained linguistic cues such as
animacy, case marking, word order etc. mediate sentence comprehension across languages. It goes beyond the
Competition Model in positing that the use of these cues follows from an underlying functional principle: to find the
25
actor (the participant primarily responsible for the state of affairs) of a given proposition as quickly and
unambiguously as possible. This processing strategy likely follows from independent principles of goal-directed
action and possibly builds on the self vs. other distinction, with the self as the actor prototype. It is also compatible
with a range of typological generalisations including patterns of language change.
Competition for actorhood accounts for a range of phenomena that purely quantitative cue variation cannot
derive: changes in cue strength over the course of a sentence; qualitatively different brain responses to similar
phenomena in different languages; qualitatively similar brain responses to cues of vastly differing strengths in
different languages. These observations are based on a range of ERP studies conducted in typologically diverse
languages.
While previous versions of the eADM essentially provided a descriptive framework which could derive
these cross-linguistic results, the eADM 4.0 (Bornkessel-Schlesewsky & Schlesewsky, in preparation) spells out the
consequences of actor-oriented comprehension for the processing architecture as a whole, from word category
recognition to integration into the broader discourse. Salient features include:
� no lexical specification of word categories
� representation of word and sentence meaning via category neutral actor/action schemas
� tight integration between sentence and discourse level (reference specification), while maintaining
qualitative distinctions between the two
� clear specification of the relation between actor and "subject" (privileged syntactic argument); subject as a
cataphoric discourse relation
I will show how these architectural assumptions (a) derive existing ERP data; (b) suggest very general (and unified)
interpretations of language-related ERP components such as the N400 and P600; (c) make predictions for
neuroanatomy.
Additional Discussion Points for Wednesday
Compare/contrast Kempen’s unification model and MacWhinney’s competition model
Relation between actor/action schemas and SemRep?
Is actorhood really enough to serve as a backbone of the whole range of linguistic comprehension? Cf.
Kemmerer.
How do the ERP components/language processing steps relate to other non-linguistic cognitive capacities?
How can ERP data best be integrated with fMRI data?
Barres: Viewing Template Construction Grammar as a Model of Comprehension The talk will offer a comprehension view of TCG in relation to eADM; giving a dual path model in relation to
aphasia involving the deep semantics of world knowledge versus the semantic categories in constructions in building
a SemRep from an utterance.
Additional Discussion Points for Wednesday
Relation to Role and Reference grammar.
Relation to FCG
Compare/contrast the tight coupling of X-schemas into ECG
Developing a Workspace to support this kind of modeling.
26
Grosvald: Dissociating linguistic and non-linguistic gesture processing: An ERP study of
American Sign Language A fundamental advance in our understanding of human language would come from a detailed account of how
non-linguistic and linguistic information is differentiated in real time by language users. To explore this issue, we
targeted the N400, an ERP component known to be sensitive to semantic context. Deaf signers saw 120 American
Sign Language sentences, each consisting of a sentence frame (e.g. BOY SLEEP IN HIS) completed by one of four
items: a semantically congruent sign (e.g. BED), a semantically incongruent sign (e.g. LEMON), a pseudo-sign
(phonologically legal but non-lexical form), or a non-linguistic grooming gesture (e.g. the performer scratching her
face). To the extent that semantic processing is similar for spoken and signed language, a larger N400 response is
expected in the semantically congruent and pseudo-sign contexts than for the semantically congruent sentences. The
response to the non-linguistic grooming gestures is of particular interest: if processed linguistically during early
processing stages, such items might also be expected to elicit an N400-like component. On the other hand, a
different response should be expected if such items are rejected as non-linguistic at early stages of processing.
We found significant N400-like responses in the incongruent and pseudo-sign contexts while in contrast, the
gestures elicited a large positive-going waveform. These findings suggest that deaf signers can quickly detect and
disregard non-linguistic manual actions during real time language processing. Moreover, they represent an important
step in understanding the relationship between the processing of language and of human actions in general, and offer
information about the time-course and neural topography of these processes.
Additional Discussion Points for Wednesday
Relation to use of ERPs in the study of spoken language.
What is specific to language versus generic to action perception?
Relation to Cartmill’s presentation.
Challenges for comparing speech and sign in seeking neural correlates.
Cartmill: From ape and human gesture to language Gesture use in human and non-human primates; imitation and ritualization of signs; sensitivity to social contexts.
The relation between gesture and speech in human infants. The role of human gesture in grounding mental
representation in action. Additional Discussion Points for Wednesday
Differences between wild and human-reared apes
Apes' responding to and manipulating others' attentional states
Semantics of gestures and their use as symbols
Relation to Lee’s SemRep
Challenge of creating databases for gestural data and configuring them as targets for modeling?
Schilling: Recruitment of Grounded Internal Models in Action, Cognition and Language Following the behavior-based approach, we believe that behavior, which is usually and sensibly being described
on a global level, emerges from the cooperation of small neuronal behavioral elements characterized at a lower level
of description without requiring a central superior controller. We started this approach by concentrating on a fairly
27
complex behavior, hexapod walking, and are now continuing this approach by treating phenomena usually attributed
to cognitive systems.
Based on a behaviour-based system we are now developing a system showing cognitive abilities while still
strongly relying on reactive elements already given. To be able to expand the Walknet for gaining properties
attributed to cognitive systems, we require the introduction of further elements. Essential is the introduction of a
manipulable body model that allows the system to test the outcome of a given behavior via internal simulation, i.e.
without explicitly performing a, possibly risky, behavior. The next step concerns a small network that allows the
system to “concentrate” on specific, randomly selected behavioral elements, the effect of which can then be tested
via internal simulation. In this way, the system is creative and is able to plan ahead, two essential properties for a
system called “cognitive”.
Finally, this model shall be extended towards language. Language recruits the underlying system and language
understanding utilizes the internal simulation. The different levels of representation appear to be tightly connected.
Therefore, on the one hand, the structure of actions shows up in how language is structured. And, on the other hand,
language information can directly be integrated into a grounded conceptual system. In our approach, we want to
connect these different levels of representations (actions given as dynamic networks and language described through
constructions) and apply them in a communicative scenario.
Additional Discussion Points for Wednesday
How does language reflect the structure of action?
Connection to Kemmerer: Structure of Actions - structure of verbs
How are different types of knowledge integrated (e.g., information from language and conceptual knowledge)?
How can action controller architectures be adapted for embodied construction grammar?
Links between ECG and TCG and FCG?
Possible links to Kemmerer
MacWhinney: The Competition Model of language processing Progress on the Unified Model is still a work in progress. The top level description is intended to bring together
existing theories into neurolinguistic contact. Modeling work is focused on the lexicon (DevLex) and syntax (item-
based patterns); non-modeling work addresses the theory of perspective. The approach views Broca's area as gating
the ability of the lexicon to trigger motor commands.
Additional Discussion Points for Wednesday
Relation to Kemmerer’s paper?
Address Bornkessel’s 2006 comments on the competition model: On-line versus off-line; one language at a time
versus bilingual.
What does leaning of bilinguals suggest for the overall process of language acquisition and comprehension?
What are the neurolinguistic evidence and implications of the suggested models, especially in relation to other
neurolinguistics researchers of the Workshop?
The CHILDES database for language acquisition as a paradigm for behavioral databases. Issue: How are data to
be summarized into a form suitable for modeling?
28
Burns: Building a Breakthrough Machine Scientific breakthroughs are both elusive and massively significant to humanity. Given that the knowledge
architecture of scientific work is itself evolving based on developments in informatics, computing, text mining and
automated acquisition methods, can we re-examine how this architecture could be designed to understand what
constitutes a scientific breakthrough? Could we even design this architecture to generate breakthroughs? I will
describe a preliminary initiative challenge program at USC's Information Sciences Institute and invite attendees to
contribute and participate. This will involve worked examples from neuroscience based on neural connectivity data
that makes use of methods for extracting information from the full-text of neuroscience publications and reasoning
over that knowledge in an online system.
Session 4
Bohland: Informatics efforts to link gene expression data to speech/language brain systems The architecture of language-related brain systems can be described at
multiple scales, from genes and gene products that guide brain development and provide the molecular signatures
of populations of neurons to the circuits that comprise functional subsystems. While much of our neurobiologically
based experimental data related to language comes from neuroimaging or lesion studies, new efforts are now
enabling connections to be made between genes and gene sets and the higher-level organization of these brain
systems. I will describe such efforts in mouse, and now human, and discuss a new related web database aimed at
neurolinguistics researchers that we are developing. I will also discuss the issue of brain atlas concordance, which is
important in unifying diverse human brain datasets.
Additional Discussion Points for Wednesday
How does this new resource complement existing resources (classical atlases, connectivity data, activation foci
data) and what are the potential integration points between them?
How can the computational modeler make use of these new resources?
What would go in a Gene Expression SED?
Relation to Marques on the linkage of Brain Navigator to the Allen Brain Atlas.
Relation to Bota on comparison of human and macaque data.
Miikkulainen: Language and neural networks The GLIDES model, a neural network architecture that shows how the symbol-grounding problem can be solved through
learned relationships between simple visual scenes and linguistic descriptions. DISCERN, a connectionist model of human story
processing, as a framework for placing sentences in a larger context by putting together different memory systems.
Additional Discussion Points for Wednesday
Relating the GLIDES model to Lee’s work.
Using DISCERN to frame broader discussion of the role of context in analyzing sentences.
What kind of neural network (subsymbolic) modeling techniques can be used for implementing models of
Construction Grammar (e.g. TCG, FCG) and other approaches (e.g. the Unification model)?
How can those applied neural network components be linked to neuro-physiological circuitry of the brain?
29
Oztop: From the concept of ‘self-observation for mirror neuron development’ to imitation and
robot skill transfer The talk will introduce the self-observation mechanism for visuomotor development in the context of mirror
neuron formation and imitation learning. Self-observation allows a system to develop internal models than can help
predict future and eliminate delays in sensory processing. There is a range of evidence showing that our brain uses
this strategy elegantly for dealing with novel tools, agency detection and body image formation. As a utilization of
this capacity of the human brain, recently we proposed a skill transfer paradigm for artificial agents. Primates, in
particular humans, are very adept at learning to use tools. The paradigm uses this sensorimotor learning capacity to
obtain agent behaviors, which otherwise would require manual programming by experts. The idea is to consider the
target agent as a tool that can be controlled by a human. Provided with an intuitive interface for controlling the agent
(e.g. a robot), the human learns to perform a given task using the agent. This is akin to the stage where a beginner is
learning to drive a car. The key point in the proposed paradigm is that the human is included in the real-time control
loop, therefore his sensorimotor learning mechanisms is fully engaged allowing the human brain to act as an
adaptive controller to accomplish a given task. Once the control proficiency has been attained by the human, then
obtaining an autonomous controller boils down to reverse engineering the control policy established by the human
brain. After introducing and exemplifying the paradigm, I will present some ideas on the brain areas that might
mediate the capacity of humans to subsume an external agent into the body schema. Finally, the fact that these areas
can be considered as parts of the mirror system will be elaborated and linked back to the mirror neurons and their
possible function in the human and non-human primates.
Additional Discussion Points for Wednesday
Relation to Bonaiuto. Affordances.
Relation to Demiris.
Issues of the databases required to support such modeling.
Linking Agency (in neuroscience) with the Subject (linguistics) of a sentence
What do we need to make an artificial system (i.e. robot) build up a (proto)language?
How to bridge the gap between sensorimotor experiences of an agent with the symbols of the language?
Gasser: Ontogenetic Ritualization in nonhuman primates Ontogenetic ritualization is a process whereby an action between two conspecifics becomes replaced by a
communicative symbol derived from aspects of the action episode:
• Individual A performs behavior X.
• Individual B reacts consistently with behavior Y.
• Subsequently B anticipates A's performance of the complete behavior X by performing Y after observing only
some initial segment X’ of X.
• Subsequently, A anticipates B's anticipation and produces the initial step in a ritualized form XR (waiting for a
response) in order to elicit Y.
It has been proposed to explain the development of specific gestures in great ape communities. Here, we will
describe several archetypal examples of ontogenetic ritualization. We then outline a computational model that
explains the steps to go from episodes of direct interaction to physically separated communicative acts directed
30
towards similar goals. In so doing, we will briefly review a set of models capturing primate action recognition and
behavioral control systems and suggest the relevant extensions necessary to capture the degree of learning and
flexible behavioral control that ontogenetic ritualization requires.
Additional Discussion Points for Wednesday
Developing a Workspace to support this kind of modeling.
Wood: How cognition functions in primates Humans make inferences about other individuals’ intentions and goals by evaluating their actions in relation to
the constraints imposed by the environment. This capacity enables humans to go beyond the surface appearance of
behavior to draw inferences about an individual’s mental states. In this talk, I describe experiments showing that this
ability is shared with nonhuman primates. I then describe a new project designed to characterize the development of
action comprehension by combining controlled-rearing experiments of animals with computational modeling based
on Bayesian inverse planning.
Additional Discussion Points for Wednesday
Comparison of humans and primates.
Event representation in relation to SemRep.
Limitations of simulation theory
Inferential reasoning
What developmental studies can add to such research
Fox: BrainMap, ALE and meta-analytic modeling.����Current status of the BrainMap database, ALE (Activation Likelihood Estimation) and related tools for
coordinate-based meta-analysis of neural systems. These tools allow construction of graphical models of neural
systems including regions (nodes), connections (edges), and functional characterizations. Meta-analytic models have
high intrinsic information content and also increase statistical power when used as starting models for causal and
graphical modeling techniques.
Additional Discussion Points for Wednesday:
As these meta-analytic neural models are computed from post hoc activation co-occurrence probabilities, how
closely do they reflect neural systems as defined by other methods?
Can these meta-analysis and modeling strategies be extrapolated to types of brain data other than activation foci?
What methods of data harvesting has BrainMap tried? What are the implications for adding SEDs to BODB.
Have the BrainMap methods been applied to neurolinguistics?
Can the BrainMap methods be related to comparable data from non-human primates on action, gesture, etc.?
Implications for adding a computational meta-analysis tool to BODB.�
Implications for adding SEDs to BODB.
Nielsen: The Brede database and new tools for meta-analysis This provides an introduction to the Brede tools for handling and analyzing the data that is associated a
neuroimaging study and how it interfaces with and can be included in other databases.
Additional Discussion Points for Wednesday
31
BredeWiki versus Database. How do you deal with data quality issues in a wiki – not just the quality of data in
the system but the quality of data return to the user during search/browse/navigation of your site.
How to deal with the issues of data ownership and sharing [within or beyond brain imagers]
Link to Burns.
Relate to Marques on the use of BrainLink in Brain Navigator.
Background for the Workshop: Abstracts of Selected Papers by Participants
Arbib
Arbib, M. A., N. Tinroongroj, Plangprasopchok, A. and Bonaiuto, J.J. (2011). Brain Operating Principles
and the Brain Operation Database (BODB): A Framework for Cumulative Modeling in Systems and
Cognitive Neuroscience.
Empirical neuroscience and computational neuroscience complement each other. The former benefits from
predictions and suggestions from modeling, while model validation rests on empirical data. Unfortunately, empirical
data and models lack an integrated neuroinformatics environment, yielding differences in data representation that
impede the computational integration of diverse data. There are no adequate tools for searching experiments and
protocols relevant to testing a given set of models, or for seeking models that can yield predictions relevant to a
certain body of experiments. We here introduce the Brain Operation Database, BODB, which is designed to provide
an environment in which modelers and experimentalists can work together by making use of their shared empirical
data, models and know-how. Our work in BODB uses the structural ontology of nested brain regions but our
contribution to neuroinformatics ontology is to introduce the complementary functional ontology of Brain Operating
Principles (BOPs). These facilitate federation by setting forth functional principles that can structure both models
and observed neural function. BODB is structured around three information entities – summaries of empirical data,
models and summaries of simulation results, and Brain Operating Principles (BOPs) – along with structures that
enhance entry searching and model comparison
Arbib, M. A. (2010). Mirror System Activity for Action and Language is Embedded in the Integration of
Dorsal & Ventral Pathways. Brain and Language 112(1): 12-24.
We develop the view that the involvement of mirror neurons in embodied experience grounds brain structures
that underlie language, but that many other brain regions are involved. We stress the cooperation between the dorsal
and ventral streams in praxis and language. Both have perceptual and motor schemas but the perceptual schemas in
the dorsal path are affordances linked to specific motor schemas for detailed motor control, whereas the ventral path
supports planning and decision making. This frames the hypothesis that the mirror system for words evolved from
the mirror system for actions to support words-as-phonological-actions, with semantics provided by the linkage to
neural systems supporting perceptual and motor schemas. We stress the importance of computational models which
can be linked to the parametric analysis of data and conceptual analysis of these models to support new patterns of
understanding of the data. In the domain of praxis, we assess the FARS model of the canonical system for grasping,
the MNS models for the mirror system for grasping, and the Augmented Competitive Queuing model that extends
the control of action to the opportunistic scheduling of action sequences and also offers a new hypothesis on the role
of mirror neurons in self action. Turning to language, we use Construction Grammar as our linguistic framework to
32
get beyond single words to phrases and sentences, and initiate analysis of what brain functions must complement
mirror systems to support this functionality.
Arbib, M. A. (2011). From Mirror Neurons to Complex Imitation in the Evolution of Language and Tool
Use. Annual Review of Anthropology 40.
The mirror system hypothesis suggests that evolution expanded a basic mirror system for grasping, in concert
with other brain regions first to support simple imitation (shared with the common ancestor of humans and great
apes) and thence to complex imitation (unique to the hominin line), which includes overimitation, the apparent
drawbacks of which are in fact essential to human skill transmission. These advances in praxis supported the
emergence of pantomime and thence protosign and protospeech. This capacity, we claim, was adequate for cultural
evolution to then yield language. We argue that Oldowan tool making corresponds to simple imitation and ape
gestural communication and Acheulean tool making corresponds to complex imitation and protolanguage, whereas
the explosion of innovations in tool making and social organization of the past 100,000 years correlates with the
emergence of language. Care is taken, however, to distinguish brain mechanisms for praxis from those supporting
language.
Arbib, M. A., J. Bonaiuto, et al. (2009). Tool use and the distalization of the end-effector. Psychol Res
73(4): 441-62.
We review recent neurophysiological data from macaques and humans suggesting that the use of tools extends
the internal representation of the actor’s hand, and relate it to our modeling of the visual control of grasping. We
introduce the idea that, in addition to extending the body schema to incorporate the tool, tool use involves
distalization of the end-effector from hand to tool. Different tools extend the body schema in different ways, with a
displaced visual target and a novel, task-specific processing of haptic feedback to the hand. This distalization is
critical in order to exploit the unique functional capacities engendered by complex tools.
Aziz-Zadeh
Aziz-Zadeh L, Sheng T, Gheytanchi A (2010) Common Premotor Regions for the Perception and
Production of Prosody and Correlations with Empathy and Prosodic Ability. PLoS ONE 5(1): e8759.
doi:10.1371/journal.pone.0008759
Prosody, the melody and intonation of speech, involves the rhythm, rate, pitch and voice quality to relay
linguistic and emotional information from one individual to another. A significant component of human social
communication depends upon interpreting and responding to another person’s prosodic tone as well as one’s own
ability to produce prosodic speech. However there has been little work on whether the perception and production of
prosody share common neural processes, and if so, how these might correlate with individual differences in social
ability. The production of prosody is well known to be a specialization of the premotor cortex, in particular the
inferior frontal gyrus (IFG), with emotional prosody more strongly activating the right hemisphere and linguistic
prosody more strongly activating the left hemisphere [1,2]. Research on the perception of prosody has largely
focused on the right temporal lobe. However, despite this emphasis, there is some indication that the premotor
cortex may also be involved [1,3,4]. Nevertheless, premotor contributions to prosody perception have not been well
studied. There is limited evidence that there may be common frontal areas active for both the perception and
production of prosody; patients with lesions to frontal areas seem to have difficulty with both the perception and
33
production of prosody [2]. However, these lesions are often very large and it is difficult to discern if the same brain
areas are utilized in the two tasks. If the same areas were to be involved, it may indicate that, at least under some
circumstances, the acoustic signals from another person’s prosodic speech are transformed into articulatory signals
in order to understand prosodic meaning. That is, it may imply that in order to understand someone else’s prosodic
intonation, we may utilize our own motor representations of how we would produce the given intonation.
Aziz-Zadeh, Wilson, Rizzolatti & Iacoboni (2006): Congruent Embodied Representations for Visually
Presented Actions and Linguistic Phrases Describing Actions, Current Biology 16, 1–6, September 19, 2006.
The thesis of embodied semantics holds that conceptual representations accessed during linguistic processing are,
in part, equivalent to the sensory-motor representations required for the enactment of the concepts. Here, using
fMRI, we tested the hypothesis that areas inhuman premotor cortex that respond both to the execution and
observation of actions—mirror neuron areas – are key neural structures in these processes. Participants observed
actions and read phrases relating to foot, hand, or mouth actions. In the premotor cortex of the left hemisphere, a
clear congruence was found between effector-specific activations of visually presented actions and of actions
described by literal phrases. These results suggest a key role of mirror neuron areas in the re-enactment of sensory-
motor representations during conceptual processing of actions invoked by linguistic stimuli.
Aziz-Zadeh, Koski, Zaidel, Mazziotta & Iacoboni (2006): Lateralization of the Human Mirror Neuron
System
A cortical network consisting of the inferior frontal, rostral inferior parietal, and posterior superior temporal
cortices has been implicated in representing actions in the primate brain and is critical to imitation in humans. This
neural circuitry may be an evolutionary precursor of neural systems associated with language. However, language is
predominantly lateralized to the left hemisphere, whereas the degree of lateralization of the imitation circuitry in
humans is unclear. We conducted a functional magnetic resonance imaging study of imitation of finger movements
with lateralized stimuli and responses. During imitation, activity in the inferior frontal and rostral inferior parietal
cortex, although fairly bilateral, was stronger in the hemisphere ipsilateral to the visual stimulus and response hand.
This ipsilateral pattern is at variance with the typical contralateral activity of primary visual and motor areas.
Reliably increased signal in the right superior temporal sulcus (STS) was observed for both left-sided and right-
sided imitation tasks, although subthreshold activity was also observed in the left STS. Overall, the data indicate that
visual and motor components of the human mirror system are not left-lateralized. The left hemisphere superiority for
language, then, must be have been favored by other types of language precursors, perhaps auditory or multimodal
action representations.
Lisa Aziz-Zadeh, Antonio Damasio, Embodied semantics for actions: Findings from functional brain
imaging Journal of Physiology - Paris 102 (2008) 35–39
The theory of embodied semantics states that concepts are represented in the brain within the same sensory-
motor circuitry in which the enactment of that concept relies. For example, the concept of ‘‘grasping” would be
represented in sensory-motor areas that represent grasping actions; the concept of ‘‘kicking” would be represented
by sensory-motor areas that control kicking actions; and so forth. This theory indicates that the secondary sensory-
motor areas, which are known to be involved in mental simulation of world experiences (Rizzolatti and Craighero,
2004), may be responsible for the representation of concepts (Gallese and Lakoff, 2005). Instead, primary sensory-
34
motor areas are presumed to be inhibited, in order to distinguish between a general representation of a given concept
and its actualization. This theory has been extended by Lakoff and his colleagues to include metaphors. Thus the
phrase ‘‘kick off the year” would also involve the motor representations related to kicking, just as the phrase ‘‘grasp
the explanation” would involve motor representations related to control of the hand (Lakoff and Johnson, 1999).
Ideas along similar lines have been discussed previously by other groups (for example, Barsalou, 1999; Damasio,
1989; Damasio and Tranel, 1993; Feldman and Narayanan, 2004; Glenberg and Kaschak, 2002; Pulvermuller, 2005;
Pulvermuller et al., 2005). While embodied semantics would apply to all concepts, (i.e., concepts associated with
visual movement would be represented by visual motion areas such as MT, etc.), most efforts to date have been
placed on investigating the neural correlates of concepts associated with actions (kicking, grasping, biting, etc.).
Here we review first some neuropsychological literature investigating the link between language and motor
processing, then move to the link between embodied semantics for language and the mirror neuron system and fMRI
data on embodied semantics for actions. Next we consider representations for language and for actions in BA 44,
and finally embodied semantics for metaphors.
Aziz-Zadeh, Lisa, Fiebach, Christian J., Naranayan, Srini, Feldman, Jerome, Dodge, Ellen and Ivry,
Richard B. (2007) 'Modulation of the FFA and PPA by language related to faces and places', Social
Neuroscience, 1 – 10
Does sentence comprehension related to faces modulate activity in the fusiform face area (FFA) and does
sentence comprehension related to places modulate activity in the parahippocampal place area (PPA)? We
investigated this question in an fMRI experiment. Participants listened to sentences describing faces, places, or
objects, with the latter serving as a control condition. In a separate run, we localized the FFA and PPA in each
participant using a perceptual task. We observed a significant interaction between the region of interest (FFA vs.
PPA) and sentence type (face vs. place). Activity in the left FFA was modulated by face sentences and in the left
PPA was modulated by place sentences. Surprisingly, activation in each region of interest was reduced when
listening to sentences requiring semantic analysis related to that region’s domain specificity. No modulation was
found in the corresponding right hemisphere ROIs. We conclude that processing sentences may involve inhibition of
some visual processing areas in a content-specific manner. Furthermore, our data indicate that this semantic-based
modulation is restricted to the left hemisphere. We discuss how these results may constrain neural models of
embodied semantics.
Bohland
The Brain Atlas Concordance Problem: Quantitative Comparison of Anatomical Parcellations
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0007200
Many neuroscientific reports reference discrete macro-anatomical regions of the brain which were delineated
according to a brain atlas or parcellation protocol. Currently, however, no widely accepted standards exist for
partitioning the cortex and subcortical structures, or for assigning labels to the resulting regions, and many
procedures are being actively used. Previous attempts to reconcile neuroanatomical nomenclatures have been largely
qualitative, focusing on the development of thesauri or simple semantic mappings between terms. Here we take a
fundamentally different approach, discounting the names of regions and instead comparing their definitions as
spatial entities in an effort to provide more precise quantitative mappings between anatomical entities as defined by
35
different atlases. We develop an analytical framework for studying this brain atlas concordance problem, and apply
these methods in a comparison of eight diverse labeling methods used by the neuroimaging community. These
analyses result in conditional probabilities that enable mapping between regions across atlases, which also form the
input to graph-based methods for extracting higher-order relationships between sets of regions and to procedures for
assessing the global similarity between different parcellations of the same brain. At a global scale, the overall results
demonstrate a considerable lack of concordance between available parcellation schemes, falling within chance levels
for some atlas pairs. At a finer level, this study reveals spatial relationships between sets of defined regions that are
not obviously apparent; these are of high potential interest to researchers faced with the challenge of comparing
results that were based on these different anatomical models, particularly when coordinate-based data are not
available. The complexity of the spatial overlap patterns revealed points to problems for attempts to reconcile
anatomical parcellations and nomenclatures using strictly qualitative and/or categorical methods. Detailed results
from this study are made available via an interactive web site at http://obart.info.
An anatomic gene expression atlas of the adult mouse brain
http://www.nature.com/neuro/journal/v12/n3/abs/nn.2281.html?lang=en
Studying gene expression provides a powerful means of understanding structure-function relationships in the
nervous system. The availability of genome-scale in situ hybridization datasets enables new possibilities for
understanding brain organization based on gene expression patterns. The Anatomic Gene Expression Atlas (AGEA)
is a new relational atlas revealing the genetic architecture of the adult C57Bl/6J mouse brain based on spatial
correlations across expression data for thousands of genes in the Allen Brain Atlas (ABA). The AGEA includes
three discovery tools for examining neuroanatomical relationships and boundaries: (1) three-dimensional
expression-based correlation maps, (2) a hierarchical transcriptome-based parcellation of the brain and (3) a facility
to retrieve from the ABA specific genes showing enriched expression in local correlated domains. The utility of this
atlas is illustrated by analysis of genetic organization in the thalamus, striatum and cerebral cortex. The AGEA is a
publicly accessible online computational tool integrated with the ABA (http://mouse.brain-map.org/agea).
Clustering of spatial gene expression patterns in the mouse brain and comparison with classical
neuroanatomy
http://www.sciencedirect.com/science/article/pii/S1046202309002035
Spatial gene expression profiles provide a novel means of exploring the structural organization of the brain.
Computational analysis of these patterns is made possible by genome-scale mapping of the C57BL/6J mouse brain
in the Allen Brain Atlas. Here we describe methodology used to explore the spatial structure of gene expression
patterns across a set of 3041 genes chosen on the basis of consistency across experimental observations (N = 2). The
analysis was performed on smoothed, co-registered 3D expression volumes for each gene obtained by aggregating
cellular resolution image data. Following dimensionality and noise reduction, voxels were clustered according to
similarity of expression across the gene set. We illustrate the resulting parcellations of the mouse brain for different
numbers of clusters (K) and quantitatively compare these parcellations with a classically-defined anatomical
reference atlas at different levels of granularity, revealing a high degree of correspondence. These observations
suggest that spatial localization of gene expression offers substantial promise in connecting knowledge at the
molecular level with higher-level information about brain organization.
36
Neural Representations and Mechanisms for the Performance of Simple Speech Sequences Jason W.
Bohland, Daniel Bullock, and Frank H. Guenther, Journal of Cognitive Neuroscience 22:7, pp. 1504–1529
Here we present a neural model that describes how the brain may represent and produce sequences of simple,
learned speech sounds. This model addresses the question of how, using a finite inventory of learned speech motor
actions, a speaker can produce arbitrarily many utterances that fall within the phonotactic and linguistic rules of her
language. At the phonological level of representation, the model implements two complementary subsystems,
corresponding to the structure and content of planned speech utterances within a neurobiologically realistic
framework that simulates interacting cortical and subcortical structures. This phonological representation is
hypothesized to interface between the higher level conceptual and morphosyntactic language centers and the lower
level speech motor control system, which itself implements only a limited set of learned motor programs. In the
current formulation, syllable-sized representations are ultimately selected through phonological encoding, and these
activate the most appropriate sensorimotor programs, commanding the execution of the planned sound. Construction
of the model was guided by previous theoretical work as well as clinical and experimental results, most notably a
companion fMRI study (Bohland & Guenther, 2006).
Bonaiuto
Bonaiuto, J. B., E. Rosta, et al. (2007). Extending the mirror neuron system model, I : Audible actions and
invisible grasps. Biol Cybern 96: 9-38.
The paper introduces mirror neuron system II (MNS2), a new version of the MNS model (Oztop and Arbib in
Biol Cybern 87 (2):116-140, 2002) of action recognition learning by mirror neurons of the macaque brain. The new
model uses a recurrent architecture that is biologically more plausible than that of the original model. Moreover,
MNS2 extends the capacity of the model to address data on audio-visual mirror neurons and on the response of
mirror neurons when the target object was recently visible but is currently hidden.
Bonaiuto, J. B. and M. A. Arbib (2010). Extending the mirror neuron system model, II: What did I just
do? A new role for mirror neurons. Biological Cybernetics 102: 341-359.
A mirror system is active both when an animal executes a class of actions (self-actions) and when it sees another
execute an action of that class. Much attention has been given to the possible roles of mirror systems in responding
to the actions of others but there has been little attention paid to their role in self-actions. In the companion article
(Bonaiuto et al., 2007) we presented MNS2, an extension of the Mirror Neuron System (MNS) model of the monkey
mirror system trained to recognize the external appearance of its own actions as a basis for recognizing the actions
of other animals when they perform similar actions. Here we further extend the study of the mirror system by
introducing the novel hypotheses that a mirror system may additionally help in monitoring the success of a self-
action and may also be activated by recognition of one’s own apparent actions as well as efference copy from one’s
intended actions. The framework for this computational demonstration is a model of action sequencing, called
augmented competitive queuing, in which action choice is based on the desirability of executable actions. We show
how this what did I just do? function of mirror neurons can contribute to the learning of both executability and
desirability which in certain cases supports rapid reorganization of motor programs in the face of disruptions.
37
Bornkessel-Schlesewsky
Borkessel & Schlesewsky (2006) - The extended argument dependency model: a neurocognitive approach
to sentence comprehension across languages
Real-time language comprehension is a principal cognitive ability and thereby relates to central properties of the
human cognitive architecture. Yet how do the presumably universal cognitive and neural substrates of language
processing relate to the astounding diversity of human languages (over 5,000)? The authors present a neurocognitive
model of online comprehension, the extended argument dependency model (eADM), that accounts for cross-
linguistic unity and diversity in the processing of core constituents (verbs and arguments). The eADM postulates
that core constituent processing proceeds in three hierarchically organized phases: (1) constituent structure building
without relational interpretation, (2) argument role assignment via a restricted set of cross-linguistically motivated
information types (e.g., case, animacy), and (3) completion of argument interpretation using information from
further domains (e.g., discourse context, plausibility). This basic architecture is assumed to be universal, with cross-
linguistic variation deriving primarily from the information types applied in Phase 2 of comprehension. This
conception can derive the appearance of similar neurophysiological and neuroanatomical processing correlates in
seemingly disparate structures in different languages and, conversely, of cross-linguistic differences in the
processing of similar sentence structures.
Bornkessel-Schlesewsky & Schlesewsky (2009) - Minimality as vacuous distinctness: Evidence from cross-
linguistic sentence comprehension
Psycholinguistic theorising has long been shaped by the assumption that the processing system endeavours to
minimise structures/relations during online comprehension. Within the scope of a recent cross-linguistic,
neurocognitive model of sentence comprehension (Bornkessel and Schlesewsky, 2006), we also proposed that the
assumption of a very general ‘Minimality’ principle can account for a variety of psycholinguistic findings from a
range of languages. In the present paper, we review empirical evidence for this notion of Minimality, before going
on to discuss its limitations. On the basis of this discussion, we propose that, rather than constituting an independent
processing principle, Minimality should be considered a subcase of a more general requirement for sentential
constituents to be distinct from one another. We show that this notion of Minimality as Distinctness (MaD) can
straightforwardly derive a wide range of findings on cross-linguistic sentence comprehension, while additionally
serving to simplify the overall processing architecture.
Bornkessel-Schlesewsky & Schlesewsky (2009) – The role of prominence information in the real-time
comprehension of transitive constructions: A cross-linguistic approach
Approaches to language processing have traditionally been formulated with reference to general cognitive
concepts (e.g. working memory limitations) of have based their representational assumptions on concepts from
linguistic theory (e.g. structure determines interpretation). Thus, many well-established generalizations about
language that have emerged from cross-linguistic/typological research have not as yet had a major influence in
shaping ideas about online processing. Here, we examine the viability of using typologically motivated concepts to
account for phenomena in online language comprehension. In particular, we focus on the comprehension of simple
transitive sentences (i.e. sentences involving two arguments/event participants) and cross-linguistics similarities and
differences in how they are processed. We argue that incremental argument interpretation in these structures is best
38
explained with reference to a range of cross-linguistically motivated, hierarchically ordered information types
termed ‘prominence scales’ (e.g. animacy, definiteness/specificity, case marking and linear order). We show that the
assumption of prominence-based argument processing can capture a wide range of recent neurocognitive findings,
as well as deriving well-known behavioral results.
Bornkessel-Schlesewsky et al. (2011) - Think globally: Cross-linguistic variation in electrophysiological
activity during sentence comprehension
This paper demonstrates systematic cross-linguistic differences in the electrophysiological correlates of conflicts
between form and meaning (semantic reversal anomalies). These engender P600 effects in English and Dutch (e.g.
Kolk et al., 2003; Kuperberg et al., 2003), but a biphasic N400 - late positivity pattern in German (Schlesewsky and
Bornkessel-Schlesewsky, 2009), and monophasic N400 effects in Turkish (Experiment 1) and Mandarin Chinese
(Experiment 2). Experiment 3 revealed that, in Icelandic, semantic reversal anomalies show the English pattern with
verbs requiring a position-based identification of argument roles, but the German pattern with verbs requiring a
case-based identification of argument roles. The overall pattern of results reveals two separate dimensions of cross-
linguistic variation: (i) the presence vs. absence of an N400, which we attribute to cross-linguistic differences with
regard to the sequence-dependence of the form-to-meaning mapping and (ii) the presence vs. absence of a late
positivity, which we interpret as an instance of a categorisation-related late P300, and which is observable when the
language under consideration allows for a binary well-formedness categorisation of reversal anomalies. We
conclude that, rather than reflecting linguistic domains such as syntax and semantics, the late positivity vs. N400
distinction is better understood in terms of the strategies that serve to optimise the form-to-meaning mapping in a
given language.
Bornkessel-Schlesewsky et al. (2010) - Prominence vs. aboutness in sequencing: A functional distinction
within the left inferior frontal gyrus
Prior research on the neural bases of syntactic comprehension suggests that activation in the left inferior frontal
gyrus (lIFG) correlates with the processing of word order variations. However, there are inconsistencies with respect
to the specific subregion within the IFG that is implicated by these findings: the pars opercularis or the pars
triangularis. Here, we examined the hypothesis that the dissociation between pars opercularis and pars triangularis
activation may reflect functional differences between clause-medial and clause-initial word order permutations,
respectively. To this end, we directly compared clause-medial and clause-initial object-before-subject orders in
German in a within-participants, event-related fMRI design. Our results showed increased activation for object-
initial sentences in a bilateral network of frontal, temporal and subcortical regions. Within the lIFG, posterior and
inferior subregions showed only a main effect of word order, whereas more anterior and superior subregions showed
effects of word order and sentence type, with higher activation for sentences with an argument in the clause-initial
position. These findings are interpreted as evidence for a functional gradation of sequence processing within the left
IFG: posterior subportions correlate with argument prominence-based (local) aspects of sequencing, while anterior
subportions correlate with aboutness-based aspects of sequencing, which are crucial in linking the current sentence
to the wider discourse. This proposal appears compatible with more general hypotheses about information
processing gradients in prefrontal cortex (Koechlin & Summerfield, 2007).
39
Bota
For the Talk:
Brain architecture management system
http://www.springerlink.com/content/w158346256g71u47/
Abstract: The nervous system can be viewed as a biological computer whose genetically determined
macrocircuitry has two basic classes of parts: gray matter regions interconnected by fiber pathways. We describe
here the basic features of an online knowledge management system for storing and inferring relationships between
data about the structural organization of nervous system circuitry. It is called the Brain architecture management
system (BAMS; http://brancusi.usc.edu/bkms) and it stores and analyzes data specifically concerned with
nomenclature and its hierarchical taxonomy, with axonal connections between regions, and with the neuronal cell
types that form regions and fiber pathways.
Bota M. & Arbib. M.A. (2004), Neuroinformatics 2(1): 19-58. Language evolution: neural homologies and
neuroinformatics
http://www.sciencedirect.com/science/article/pii/S0893608003002399
Abstract: This paper contributes to neurolinguistics by grounding an evolutionary account of the readiness of the
human brain for language in the search for homologies between different cortical areas in macaque and human. We
consider two hypotheses for this grounding, that of Aboitiz and García [Brain Res. Rev. 25 (1997) 381] and the
Mirror System Hypothesis of Rizzolatti and Arbib [Trends Neurosci. 21 (1998) 188] and note the promise of
computational modeling of neural circuitry of the macaque and its linkage to analysis of human brain imaging data.
In addition to the functional differences between the two hypotheses, problems arise because they are grounded in
different cortical maps of the macaque brain. In order to address these divergences, we have developed several
neuroinformatics tools included in an on-line knowledge management system, the NeuroHomology Database, which
is equipped with inference engines both to relate and translate information across equivalent cortical maps and to
evaluate degrees of homology for brain regions of interest in different species.
For future discussion, add:
From gene networks to brain networks
http://www.nature.com/neuro/journal/v6/n8/abs/nn1096.html
Abstract: The brain's structural organization is so complex that 2,500 years of analysis leaves pervasive
uncertainty about (i) the identity of its basic parts (regions with their neuronal cell types and pathways
interconnecting them), (ii) nomenclature, (iii) systematic classification of the parts with respect to topographic
relationships and functional systems and (iv) the reliability of the connectional data itself. Here we present a
prototype knowledge management system (http://brancusi.usc.edu/bkms/) for analyzing the architecture of brain
networks in a systematic, interactive and extendable way. It supports alternative interpretations and models, is based
on fully referenced and annotated data and can interact with genomic and functional knowledge management
systems through web services protocols.
Swanson L.W. & Bota M. (2010), Proc Natl Acad Sci U S A. Foundational model of structural connectivity in the nervous system with a schema for wiring diagrams, connectome, and basic plan architecture.
http://www.pnas.org/content/107/48/20610.long
40
The nervous system is a biological computer integrating the body's reflex and voluntary environmental
interactions (behavior) with a relatively constant internal state (homeostasis)—promoting survival of the individual
and species. The wiring diagram of the nervous system's structural connectivity provides an obligatory foundational
model for understanding functional localization at molecular, cellular, systems, and behavioral organization levels.
This paper provides a high-level, downwardly extendible, conceptual framework—like a compass and map—for
describing and exploring in neuroinformatics systems (such as our Brain Architecture Knowledge Management
System) the structural architecture of the nervous system's basic wiring diagram. For this, the Foundational Model of
Connectivity's universe of discourse is the structural architecture of nervous system connectivity in all animals at all
resolutions, and the model includes two key elements—a set of basic principles and an internally consistent set of
concepts (defined vocabulary of standard terms)—arranged in an explicitly defined schema (set of relationships
between concepts) allowing automatic inferences. In addition, rules and procedures for creating and modifying the
foundational model are considered. Controlled vocabularies with broad community support typically are managed
by standing committees of experts that create and refine boundary conditions, and a set of rules that are available on
the Web.
Burns
Knowledge management of the neuroscientific literature: the data model and underlying strategy of the
NeuroScholar system
http://rstb.royalsocietypublishing.org/content/356/1412/1187.short
This paper describes the underlying strategy and system's design of a knowledge management system for the
neuroscientific literature called ‘NeuroScholar’. The problem that the system is designed to address is to delineate
fully the neural circuitry involved in a specific behaviour. The use of this system provides experimental
neuroscientists with a new method of building computational models (‘knowledge models’) of the contents of the
published literature. These models may provide input for analysis (conceptual or computational), or be used as
constraint sets for conventional neural modelling work. The underlying problems inherent in this approach, the
general framework for the proposed solution, the practical issues concerning usage of the system and a detailed,
technical account of the system are described. The author uses a widely used software specification language (the
Universal Modelling Language) to describe the design of the system and present examples from published work
concerned with classical eyeblink conditioning in the rabbit.
Tools and approaches for the construction of knowledge models from the neuroscientific literature
http://www.springerlink.com/content/8t786482928v5q77/
Within this paper, we describe a neuroinformatics project (called NeuroScholar, http://www.neuroscholar.org/)
that enables researchers to examine, manage, manipulate, and use the information contained within the published
neuroscientific literature. The project is built within a multi-level, multi-component framework constructed with the
use of software engineering methods that themselves provide code-building functionality for neuroinformaticians.
We describe the different software layers of the system. First, we present a hypothetical usage scenario illustrating
how NeuroScholar permits users to address largescale questions in a way that would otherwise be impossible. We
do this by applying NeuroScholar to a real-world neuroscience question: How is stress-related information
processed in the brain? We then explain how the overall design of NeuroScholar enables the system to work and
41
illustrate different components of the user interface. We then describe the knowledge management strategy we use
to store interpretations. Finally, we describe the software engineering framework we have devised (called the View-
Primitive-Data Model framework, [VPDMf]) to provide an open-source, accelerated software development
environment for the project. We believe that NeuroScholar will be useful to experimental neuroscientists by helping
them interact with the primary neuroscientific literature in a meaningful way, and to neuroinformaticians by
providing them with useful, affordable software engineering tools.
Intelligent Approaches to Mining the Primary Research Literature: Techniques, Systems, and Examples
http://www.springerlink.com/content/u5h203h351q74887/
In this chapter, we describe how creating knowledge bases from the primary biomedical literature is formally
equivalent to the process of performing a literature review or a ‘research synthesis’. We describe a principled
approach to partitioning the research literature according to the different types of experiments performed by
researchers and how knowledge engineering approaches must be carefully employed to model knowledge from
different types of experiment. The main body of the chapter is concerned with the use of text mining approaches to
populate knowledge representations for different types of experiment. We provide a detailed example from
neuroscience (based on anatomical tract-tracing experiments) and provide a detailed description of the methodology
used to perform the text mining itself (based on the Conditional Random Fields model). Finally, we present data
from textmining experiments that illustrate the use of these methods in a real example. This chapter is designed to
act as an introduction to the field of biomedical text-mining for computer scientists who are unfamiliar with the way
that biomedical research uses the literature.
Cartmill
Cartmill & Byrne (2010): Semantics of primate gestures: intentional meanings of orangutan gestures
Great ape gesture has become a research topic of intense interest, because its intentionality and flexibility suggest
strong parallels to human communication. Yet the fundamental question of whether an animal species’ gestures
carry specific meanings has hardly been addressed. We set out a systematic approach to studying intentional
meaning in the gestural communication of non-humans and apply it to a sample of orangutan gestures. We propose
that analysis of meaning should be limited to gestures for which (1) there is strong evidence for intentional
production and (2) the recipient’s final reaction matches the presumed goal of the signaller, as determined
independently. This produces a set of successful instances of gesture use, which we describe as having goal–
outcome matches. In this study, 28 orangutans in three European zoos were observed for 9 months. We
distinguished 64 gestures on structural grounds, 40 of which had frequent goal–outcome matches and could
therefore be analysed for meaning. These 40 gestures were used predictably to achieve one of 6 social goals: to
initiate an affiliative interaction (contact, grooming, or play), request objects, share objects, instigate co-locomotion,
cause the partner to move back, or stop an action. Twenty-nine of these gestures were used consistently with a single
meaning. We tested our analysis of gesture meaning by examining what gesturers did when the response to their
gesture did not match the gesture’s meaning. Subsequent actions of the gesturer were consistent with our
assignments of meaning to gestures. We suggest that, despite their contextual flexibility, orangutan gestures are
made with the expectation of specific behavioural responses and thus have intentional meanings as well as
functional consequences.
42
Cartmill & Byrne (2007): Orangutans Modify Their Gestural Signaling According to Their Audience's
Comprehension
When people are not fully understood, they persist with attempts to communicate, elaborating their speech in
order to better convey their meaning [1]. We investigated whether captive orangutans (Pongo pygmaeus and Pongo
abelii) would use analogous communicative strategies in signaling to a humanexperimenter, and whether they could
distinguish different degrees of misunderstanding. Orangutans' behavior varied according to how well they had
apparently been understood. When their aims were not met, they persisted in communicative attempts. However,
when the interlocutor appeared partially to understand their meaning, orangutans narrowed down their range of
signals, focusing on gestures already used and repeating them frequently. In contrast, when completely
misunderstood, orangutans elaborated their range of gestures, avoiding repetition of failed signals. It is therefore
possible, from communicative signals alone, to determine how well an orangutan's intended goal has been met. This
differentiation might function under natural conditions to allow an orangutan's intended goals to be understood more
efficiently. In the absence of conventional labels, communicating the fact that an intention has been somewhat
misunderstood is an important way to establish shared meaning.
Cartmill, E., Beilock, S., & Goldin-Meadow, S. (in press) A word in the hand: Action, gesture, and mental
representation in humans and non-human primates. Philosophical Transactions of the Royal Society B
(Special Issue “Action to Language”)
The movements we make with our hands both reflect our mental processes and help to shape them. Our actions
and gestures can affect our mental representations of actions and objects. In this paper, we explore the relationship
between action, gesture, and thought in both humans and non-human primates and discuss its role in the evolution of
language. Human gesture (specifically representational gesture) may provide a unique link between action and
mental representation. It is kinesthetically close to action and is, at the same time, symbolic. Non-human primates
use gesture frequently to communicate, and do so flexibly. However, their gestures mainly resemble incomplete
actions and lack the representational elements that characterize much of human gesture. Differences in the mirror
neuron system provide a potential explanation for non-human primates’ lack of representational gestures; the
monkey mirror system does not respond to representational gestures, while the human system does. In humans,
gesture grounds mental representation in action, but there is no evidence for this link in other primates. We argue
that gesture played an important role in the transition to symbolic thought and language in human evolution,
following a cognitive leap that allowed gesture to incorporate representational elements.
Cartmill, E. A., Demir, Ö. E. & Goldin-Meadow, S. (in press). Studying Gesture. In E. Hoff (Ed.) Research
Methods in Child Language: A Practical Guide, First Edition. Blackwell Publishing Ltd.
To gain a full understanding of the steps children follow in acquiring language, researchers must pay attention to
their hands as well as their mouths – that is, to gesture. We first define our methodology for studying gesture. We
then describe different types of gestures and their typical uses, and the methods by which meaning can be attributed
to gesture. We stress the importance of characterizing the relationship between gesture and speech, and illustrate
how that relationship changes over time as children’s spoken language develops. Importantly, the methods for
coding and analyzing gesture in relation to speech also change over time, and we provide examples of these
changes. We end by discussing gesture’s role in language learning and later stages of cognitive development.
43
Demiris
Demiris & Khadhouri (2008): Content-based control of goal-directed attention during human action
perception
http://www.iis.ee.ic.ac.uk/yiannis/DemirisKhadhouri08-IS.pdf
During the perception of human actions by robotic assistants, the robotic assistant needs to direct its
computational and sensor resources to relevant parts of the human action. In previous work we have introduced
HAMMER (Hierarchical Attentive Multiple Models for Execution and Recognition) (Demiris and Khadhouri,
2006), a computational architecture that forms multiple hypotheses with respect to what the demonstrated task is,
and multiple predictions with respect to the forthcoming states of the human action. To confirm their predictions, the
hypotheses request information from an attentional mechanism, which allocates the robot’s resources as a function
of the saliency of the hypotheses. In this paper we augment the attention mechanism with a component that
considers the content of the hypotheses’ requests, with respect to the content’s reliability, utility and cost. This
content-based attention component further optimises the utilisation of the resources while remaining robust to noise.
Such computational mechanisms are important for the development of robotic devices that will rapidly respond to
human actions, either for imitation or collaboration purposes.
Demiris (2007): Prediction of intent in robotics and multi-agent systems
http://www.iis.ee.ic.ac.uk/yiannis/DemirisCP07.pdf
Moving beyond the stimulus contained in observable agent behaviour, i.e. understanding the underlying intent of
the observed agent is of immense interest in a variety of domains that involve collaborative and competitive
scenarios, for example assistive robotics, computer games, robot–human interaction, decision support and intelligent
tutoring. This review paper examines approaches for performing action recognition and prediction of intent from a
multi-disciplinary perspective, in both single robot and multi-agent scenarios, and analyses the underlying
challenges, focusing mainly on generative approaches
Demiris & Simmons (2006): Perceiving the unusual: Temporal properties of hierarchical motor
representations for action perception
http://www.iis.ee.ic.ac.uk/yiannis/DemirisSimmonsNN06.pdf
Recent computational approaches to action imitation have advocated the use of hierarchical representations in the
perception and imitation of demonstrated actions. Hierarchical representations present several advantages, with the
main one being their ability to process information at multiple levels of detail. However, the nature of the
hierarchies in these approaches has remained relatively unsophisticated, and their relation with biological evidence
has not been investigated in detail, in particular with respect to the timing of movements. Following recent
neuroscience work on the modulation of the premotor mirror neuron activity during the observation of unpredictable
grasping movements, we present here an implementation of our HAMMER architecture using the minimum
variance model for implementing reaching and grasping movements that have biologically plausible trajectories.
Subsequently, we evaluate the performance of our model in matching the temporal dynamics of the modulation of
cortical excitability during the passive observation of normal and unpredictable movements of human
demonstrators.
Johnson & Demiris (2005): Perceptual Perspective Taking and Action Recognition
44
http://www.iis.ee.ic.ac.uk/yiannis/JohnsonDemiris-ijars05.pdf
Robots that operate in social environments need to be able to recognise and understand the actions of other
robots, and humans, in order to facilitate learning through imitation and collaboration. The success of the simulation
theory approach to action recognition and imitation relies on the ability to take the perspective of other people, so as
to generate simulated actions from their point of view. In this paper, simulation of visual perception is used to
recreate the visual egocentric sensory space and egocentric behaviour space of an observed agent, and through this
increase the accuracy of action recognition. To demonstrate the approach, experiments are performed with a robot
attributing perceptions to and recognising the actions of a second robot similar mechanisms for retaining different
types of visual information.
Y. Demiris and A. Meltzoff (2008) “The Robot in the Crib: A developmental analysis of imitation skills in
infants and robots”, Infant and Child Development, 17:43-53.
http://www.iis.ee.ic.ac.uk/yiannis/DemirisMeltzoff08-ICD.pdf
Interesting systems, whether biological or artificial, develop. Starting from some initial conditions, they respond
to environmental changes, and continuously improve their capabilities. Developmental psychologists have dedicated
significant effort to studying the developmental progression of infant imitation skills, because imitation underlies the
infant’s ability to understand and learn from his or her social environment. In a converging intellectual endeavor,
roboticists have been equipping robots with the ability to observe and imitate human actions because such abilities
can lead to rapid teaching of robots to perform tasks. We provide here a comparative analysis between studies of
infants imitating and learning from human demonstrators, and computational experiments aimed at equipping a
robot with such abilities. We will compare the research across the following two dimensions: (a) initial conditions -
what is innate in infants, and what functionality is initially given to robots, and (b) developmental mechanisms -
how does the performance of infants improve over time, and what mechanisms are given to robots to achieve
equivalent behavior. Both developmental science and robotics are critically concerned with: (a) how their systems
can and do go ‘beyond the stimulus’ given during the demonstration, and (b) how the internal models used in this
process are acquired over the lifetime of the system.
Fox
Publication links and pdfs may be found at: http://www.brainmap.org/
Eickhoff, S.B., Laird, A.R., Grefkes, C., Wang, L.E., Zilles, K., Fox, P.T., 2009. Coordinate-Based
Activation Likelihood Estimation Meta-Analysis of Neuroimaging Data: A Random-Effects Approach Based
on Empirical Estimates of Spatial Uncertainty. Human Brain Mapping 30, 2907- 2926.
A widely used technique for coordinate-based meta-analyses of neuroimaging data is activation likelihood
estimation (ALE). ALE assesses the overlap between foci based on modeling them as probability distributions
centered at the respective coordinates. In this Human Brain Project/Neuroinformatics research, the authors present a
revised ALE algorithm addressing drawbacks associated with former implementations. The first change pertains to
the size of the probability distributions, which had to be specified by the used. To provide a more principled
solution, the authors analyzed fMRI data of 21 subjects, each normalized into MNI space using nine different
approaches. This analysis provided quantitative estimates of between-subject and between-template variability for
16 functionally defined regions, which were then used to explicitly model the spatial uncertainty associated with
45
each reported coordinate. Secondly, instead of testing for an above-chance clustering between foci, the revised
algorithm assesses above-chance clustering between experiments. The spatial relationship between foci in a given
experiment is now assumed to be fixed and ALE results are assessed against a null-distribution of random spatial
association between experiments. Critically, this modification entails a change from fixed- to random-effects
inference in ALE analysis allowing generatization of the results to the entire population of studies analyzed. By
comparative analysis of real and simulated data, the authors showed that the revised ALE-algorithm overcomes
conceptual problems of former meta-analyses and increases the specificity of the ensuing results without loosing the
sensitivity of the original approach. It may thus provide a methodologically improved tool for coordinate-based
meta-analyses on functional imaging data.
Laird, A.R., Eickhoff, S.B., Li, K., Robin, D.A., Glahn, D.C., Fox, P.T., 2009. Investigating the Functional
Heterogeneity of the Default Mode Network Using Coordinate-Based Meta- Analytic Modeling. J Neurosci.
29, 14496-14505.
The default mode network (DMN) comprises a set of regions that exhibit ongoing, intrinsic activity in the resting
state and task-related decreases in activity across a range of paradigms. However, DMN regions have also been
reported as task-related increases, either independently or coactivated with other regions in the network. Cognitive
subtractions and the use of low-level baseline conditions have generally masked the functional nature of these
regions. Using a combination of activation likelihood estimation, which assesses statistically significant
convergence of neuroimaging results, and tools distributed with the BrainMap database, we identified core regions
in the DMN and examined their functional heterogeneity. Meta-analytic coactivation maps of task-related increases
were independently generated for each region, which included both within-DMN and non-DMN connections. Their
functional properties were assessed using behavioral domain metadata in BrainMap. These results were integrated to
determine a DMN connectivity model that represents the patterns of interactions observed in task-related increases
in activity across diverse tasks. Subnetwork components of this model were identified, and behavioral domain
analysis of these cliques yielded discrete functional properties, demonstrating that components of the DMN are
differentially specialized. Affective and perceptual cliques of the DMN were identified, as well as the cliques
associated with a reduced preference for motor processing. In summary, we used advanced coordinate-based meta-
analysis techniques to explicate behavior and connectivity in the default mode network; future work will involve
applying this analysis strategy to other modes of brain function, such as executive function or sensorimotor systems.
Neumann, J., Fox, P.T., Turner, R., Lohmann, G., 2010. Learning partially directed functional networks
from meta-analysis imaging data. Neuroimage 49 (2), 13721384.
We propose a new exploratory method for the discovery of partially directed functional networks from fMRI
meta-analysis data. The method performs structure learning of Bayesian networks in search of directed probabilistic
dependencies between brain regions. Learning is based on the co-activation of brain regions observed across several
independent imaging experiments. In a series of simulations, we !rst demonstrate the reliability of the method. We
then present the application of our approach in an extensive meta-analysis including several thousand activation
coordinates from more than 500 imaging studies. Results show that our method is able to automatically infer
Bayesian networks that capture both directed and undirected probabilistic dependencies between a number of brain
regions, including regions that are frequently observed in motor-related and cognitive control tasks.
46
Price L, Laird AR, Fox PT. (2009) Modeling Dynamic Functional Neuroimaging Data Using Structural
Equation Modeling. Structural Equation Modeling 16: 147-162.
The aims of this study were to present a method for developing a path analytic network model using data
acquired from positron emission tomography. Regions of interest within the human brain were identified through
quantitative activation likelihood estimation meta-analysis. Using this information, a “true” or population path
model was then developed using Bayesian structural equation modeling. To evaluate the impact of sample size on
parameter estimation bias, proportion of parameter replication coverage, and statistical power, a 2 group
(clinical/control) x 6 (sample size Markov chain Monte Carlo study was conducted. Results indicate that using a
sample size of less than N = 15 per group will produce parameter estimates exhibiting bias greater than 5% and
statistical power below .80.
Robinson, J.L., Laird, A.R., Glahn, D.C., Lovallo, W.R., Fox, P.T., 2010. Metaanalytic connectivity
modeling: delineating the functional connectivity of the human amygdala. Hum. Brain Mapp. 31 (2), 173184.
Functional neuroimaging has evolved into an indispensable tool for noninvasively investigating brain function. A
recent development of such methodology is the creation of connectivity models for brain regions and related
networks, efforts that have been inhibited by notable limitations. We present a new method for ascertaining
functional connectivity of specific brain structures using metaanalytic connectivity modeling (MACM), along with
validation of our method using a nonhuman primate database. Drawing from decades of neuroimaging research and
spanning multiple behavioral domains, the method overcomes many weaknesses of conventional connectivity
analyses and provides a simple, automated alternative to developing accurate and robust models of anatomically-
defined human functional connectivity. Applying MACM to the amygdala, a small structure of the brain with a
complex network of connections, we found high coherence with anatomical studies in nonhuman primates as well as
human-based theoretical models of emotive-cognitive integration, providing evidence for this novel method’s utility.
Grosvald
Grosvald, Gutierrez, & Corina (2011). American Sign Language and gesture processing in deaf signers:
An ERP study. Poster presentation, 18th Annual Meeting of the Cognitive Neuroscience Society, San
Francisco.
Articulatory properties of signed languages share much of the same anatomy and physiology as those involved in
everyday manual actions and gestures. An advance in our understanding of human language would come from a
detailed account of how linguistic and non-linguistic manual actions are differentiated in real time by language users.
To explore this issue, we investigated deaf signers’ brain response when processing ASL sentences incorporating
target items of varying semantic-linguistic status. Each sentence consisted of a frame followed by a last item
belonging to one of four categories: a semantically congruent sign completing the sentence, a semantically
incongruent sign, a non-sign (phonologically legal but non-occurring form), or a non-linguistic grooming action.
We evaluated signers’ response to these final items by examining the N400, a well-studied electrophysiological
(ERP) component known to be sensitive to lexical-semantic integration. This negative-going component is generally
seen at central and parietal scalp sites, peaking about 400 ms after the onset of the relevant stimulus (Kutas &
Hillyard, 1980).
47
Corina, Grosvald et al. (In press). Perceptual invariance or orientation specificity in American Sign
Language? Evidence from repetition priming for signs and gestures. Language and Cognitive Processes.
Repetition priming has been successfully employed to examine stages of processing in a wide variety of
cognitive domains including language, object recognition, and memory. This study uses a novel repetition priming
paradigm in the context of a categorisation task to explore early stages in the processing of American Sign Language
signs and self-grooming gestures. Specifically, we investigated the degree to which deaf signers’ and hearing
nonsigners’ perception of these linguistic or nonlinguistic actions might be differentially robust to changes in
perceptual viewpoint. We conjectured that to the extent that signers were accessing language-specific
representations in their performance of the task, they might show more similar priming effects under different
viewing conditions than hearing subjects. In essence, this would provide evidence for a visually based ‘‘lack of
invariance’’ phenomenon. However, if the early stages of visual-action processing are similar for deaf and hearing
subjects, then no such difference should be found. In both groups, we observed robust effects of viewpoint,
indicating that repetition priming for identical prime_target pairs was greater than in cases of categorisation in
which the prime and target varied in viewpoint. However, we found little evidence of group-related differences that
could be interpreted as effects of perceptual invariance. These outcomes indicate that initial stages of sign and
gesture recognition required for the categorisation of action types do not differ as a function of experience with a
signed language. Instead, our data are consistent with and extend previously described visual-perceptual studies that
have reported evidence for orientation-specific representations of human actions.
Grosvald, Lachaud & Corina (In press). Handshape monitoring: Evaluation of linguistic and perceptual
factors in the processing of American Sign Language. Language and Cognitive Processes.
We investigated the relevance of linguistic and perceptual factors to sign processing by comparing hearing
individuals and deaf signers as they performed a handshape monitoring task, a sign-language analogue to the
phoneme-monitoring paradigms used in many spoken-language studies. Each subject saw a series of brief video
clips, each of which showed either an ASL sign or a phonologically possible but non-lexical “pseudo-sign,” and
responded when the viewed action was formed with a particular handshape. Stimuli varied with respect to the
factors of Lexicality, handshape Markedness (Battison, 1978), and Type, defined according to whether the action is
performed with one or two hands and for two-handed stimuli, whether or not the action is symmetrical. Deaf signers
performed faster and more accurately than hearing non-signers, and effects related to handshape Markedness and
stimulus Type were observed in both groups. However, no effects or interactions related to Lexicality were seen. A
further analysis restricted to the deaf group indicated that these results were not dependent upon subjects’ age of
acquisition of ASL. This work provides new insights into the processes by which the handshape component of sign
forms is recognized in a sign language, the role of language experience, and the extent to which these processes may
or may not be considered specifically linguistic."
Kemmerer
Kemmerer et al. (2010) - The Two-Level Theory of verb meaning: An approach to integrating the
semantics of action with the mirror neuron system
Verbs have two separate levels of meaning. One level reflects the uniqueness of every verb and is called the root.
The other level consists of a more austere representation that is shared by all the verbs in a given class and is called
48
the event structure template. We explore the following hypotheses about how, with specific reference to the motor
features of action verbs, these two distinct levels of semantic representation might correspond to two distinct levels
of the mirror neuron system. Hypothesis 1: Root-level motor features of verb meaning are partially subserved by
somatotopically mapped mirror neurons in the left primary motor and/or premotor cortices. Hypothesis 2: Template-
level motor features of verb meaning are partially subserved by representationally more schematic mirror neurons in
Brodmann area 44 of the left inferior frontal gyrus. Evidence has been accumulating in support of the general
neuroanatomical claims made by these two hypotheses-namely, that each level of verb meaning is associated with
the designated cortical areas. However, as yet no studies have satisfied all the criteria necessary to support the more
specific neurobiological claims made by the two hypotheses-namely, that each level of verb meaning is associated
with mirror neurons in the pertinent brain regions. This would require demonstrating that within those regions the
same neuronal populations are engaged during (a) the linguistic processing of particular motor features of verb
meaning, (b) the execution of actions with the corresponding motor features, and (c) the observation of actions with
the corresponding motor features.
Kemmerer et al. (2008) - Neuroanatomical distribution of five semantic components of verbs: Evidence
from fMRI
The Simulation Framework, also known as the Embodied Cognition Framework, maintains that conceptual
knowledge is grounded in sensorimotor systems. To test several predictions that this theory makes about the neural
substrates of verb meanings; we used functional magnetic resonance imaging (fMRI) to scan subjects' brains while
they made semantic judgments involving five classes of verbs-specifically, Running verbs (e.g., run, jog, walk),
Speaking verbs (e.g., shout, mumble, whisper), Hitting verbs (e.g., hit, poke, jab), Cutting verbs (e.g., cut, slice,
hack), and Change of State verbs (e.g., shatter, smash, crack). These classes were selected because they vary with
respect to the presence or absence of five distinct semantic components-specifically, ACTION; MOTION,
CONTACT, CHANGE OF STATE, and TOOL USE. Based on the Simulation Framework, we hypothesized that
the ACTION component depends on the primary motor and premotor cortices, that the MOTION component
depends on the posterolateral temporal cortex, that the CONTACT component depends on the intraparietal sulcus
and inferior parietal lobule, that the CHANGE OF STATE component depends on the ventral temporal cortex, and
that the TOOL USE component depends on a distributed network of temporal, parietal, and frontal regions. Virtually
all of the predictions were confirmed. Taken together, these findings support the Simulation Framework and extend
our understanding of the neuroanatomical distribution of different aspects of verb meaning.
Kemmerer et al. (2011) - A Functional Role for Motor Simulation in Identifying Tools
Embodied cognition promotes the involvement of the motor system in cognitive processing, such as tool
identification. Although neuropsychological studies suggest that the motor system is not necessary for identifying
tools, it may still have a functional role in tool recognition. To test this possibility, we used a motor interference
task: Participants squeezed a rubber ball in one hand while naming pictures of tools and animals. Participants were
faster and more accurate in naming the tools that were oriented with the handle facing away from the squeezing
hand than in naming the tools that were oriented with the handle facing toward the squeezing hand. There was no
effect of orientation for animals. Given that participants simulate grasping a tool with the hand closest to the handle,
49
this result demonstrates that interfering with the ability to simulate grasping impairs tool naming and suggests that
motor simulation has a functional role in tool identification.
Kemmerer et al. (2010) - Nouns and verbs in the brain: Implications of linguistic typology for cognitive
neuroscience
In recent years, cognitive neuroscience has generated many important findings about how the noun-verb
distinction is implemented in the brain, but at the same time it has largely ignored equally important advances in
linguistic typology concerning the nature of grammatical categories. Following the lead of Evans and Levinson
(2009), we argue that typological data and theory have significant implications not only for the interpretation of
recent neuroscientific discoveries about the noun-verb distinction, but also for the direction of future neuroscientific
research on this topic.
Kemmerer et al. (in press). Behavioral patterns and lesion sites associated with impaired processing of
lexical and conceptual knowledge of actions. Cortex. [Published online.]
To further investigate the neural substrates of lexical and conceptual knowledge of actions, we administered a
battery of six tasks to 226 brain-damaged patients with widely distributed lesions in the left and right cerebral
hemispheres. The tasks probed lexical and conceptual knowledge of actions in a variety of verbal and non-verbal
ways, including naming, word-picture matching, attribute judgments involving both words and pictures, and
associative comparisons involving both words and pictures. Of the 226 patients who were studied, 61 failed one or
more of the six tasks, with four patients being impaired on the entire battery, and varied numbers of patients being
impaired on varied combinations of tasks. Overall, the 61 patients manifested a complex array of associations and
dissociations across the six tasks. The lesion sites of 147 of the 226 patients were also investigated, using formal
methods for lesion-deficit statistical mapping and power analysis of lesion overlap maps. Significant effects for all
six tasks were found in the following left- hemisphere regions: the inferior frontal gyrus; the ventral precentral gyrus,
extending superiorly into what are likely to be hand- related primary motor and premotor areas; and the anterior
insula. In addition, significant effects for 4-5 tasks were found in not only the regions just mentioned, but also in
several other left-hemisphere areas: the ventral postcentral gyrus; the supramarginal gyrus; and the posterior middle
temporal gyrus. These results converge with previous research on the neural underpinnings of action words and
concepts. However, the current study goes considerably beyond most previous investigations by providing extensive
behavioral and lesion data for an unusually large and diverse sample of brain-damaged patients, and by
incorporating multiple measures of verb comprehension. Regarding theoretical implications, the study provides new
support for the Embodied Cognition Framework, which maintains that
conceptual knowledge is grounded in sensorimotor systems.
Kempen
Vosse, Theo & Kempen, Gerard (2009). The Unification Space implemented as a localist neural net:
Predictions and error-tolerance in a constraint-based parser. Cognitive Neurodynamics, 3, 331-346.
We introduce a novel computer implementation of the Unification-Space parser (Vosse and Kempen in Cognition 75:105–143,
2000) in the form of a localist neural network whose dynamics is based on interactive activation and inhibition. The wiring of the
network is determined by Performance Grammar (Kempen and Harbusch in Verb constructions in German and Dutch.
Benjamins, Amsterdam, 2003), a lexicalist formalism with feature unification as binding operation. While the network is
50
processing input word strings incrementally, the evolving shape of parse trees is represented in the form of changing patterns of
activation in nodes that code for syntactic properties of words and phrases, and for the grammatical functions they fulfill. The
system is capable, at least qualitatively and rudimentarily, of simulating several important dynamic aspects of human syntactic
parsing, including garden-path phenomena and reanalysis, effects of complexity (various types of clause embeddings),
faulttolerance in case of unification failures and unknown words, and predictive parsing (expectation-based analysis, surprisal
effects). English is the target language of the parser described.
Vosse, Theo & Kempen, Gerard (2000). Syntactic structure assembly in human parsing: A computational
model based on competitive inhibition and a lexicalist grammar. Cognition, 75, 105-143.
We present the design, implementation and simulation results of a psycholinguistic model of human syntactic processing that
meets major empirical criteria. The parser operates in conjunction with a lexicalist grammar and is driven by syntactic
information associated with heads of phrases. The dynamics of the model are based on competition by lateral inhibition
(‘competitive inhibition’). Input words activate lexical frames (i.e. elementary trees anchored to input words) in the mental
lexicon, and a network of candidate ‘unification links’ is set up between frame nodes. These links represent tentative attachments
that are graded rather than all-or-none. Candidate links that, due to grammatical or ‘treehood’ constraints, are incompatible,
compete for inclusion in the final syntactic tree by sending each other inhibitory signals that reduce the competitor's attachment
strength. The outcome of these local and simultaneous competitions is controlled by dynamic parameters, in particular by the
Entry Activation and the Activation Decay rate of syntactic nodes, and by the Strength and Strength Build-up rate of Unification
links. In case of a successful parse, a single syntactic tree is returned that covers the whole input string and consists of lexical
frames connected by winning Unification links. Simulations are reported of a significant range of psycholinguistic parsing
phenomena in both normal and aphasic speakers of English: (i) various effects of linguistic complexity (single versus double,
center versus right-hand self-embeddings of relative clauses; the difference between relative clauses with subject and object
extraction; the contrast between a complement clause embedded within a relative clause versus a relative clause embedded within
a complement clause); (ii) effects of local and global ambiguity, and of word-class and syntactic ambiguity (including recency
and length effects); (iii) certain difficulty-of-reanalysis effects (contrasts between local ambiguities that are easy to resolve versus
ones that lead to serious garden-path effects); (iv) effects of agrammatism on parsing performance, in particular the performance
of various groups of aphasic patients on several sentence types.
Vosse, Theo & Kempen, Gerard (2009). In defense of competition during syntactic ambiguity resolution.
Journal of Psycholinguistic Research, 38, 1-9
In a recent series of publications (Traxler et al. J Mem Lang 39:558–592, 1998; Van Gompel et al. J Mem Lang 52:284–307,
2005; see also Van Gompel et al. (In: Kennedy, et al.(eds) Reading as a perceptual process, Oxford, Elsevier pp 621–648, 2000);
Van Gompel et al. J Mem Lang 45:225–258, 2001) eye tracking data are reported showing that globally ambiguous (GA)
sentences are read faster than locally ambiguous (LA) counterparts. They argue that these data rule out constraint-based models
where syntactic and conceptual processors operate concurrently and syntactic ambiguity resolution is accomplished by
competition. Such models predict the opposite pattern of reading times. However, this argument against competition is valid only
in conjunction with two standard assumptions in current constraint-based models of sentence comprehension: (1) that syntactic
competitions (e.g., Which is the best attachment site of the incoming constituent?) are pooled together with conceptual
competitions (e.g., Which attachment site entails the most plausible meaning?), and (2) that the duration of a competition is a
function of the overall (pooled) quality score obtained by each competitor. We argue that it is not necessary to abandon
competition as a successful basis for explaining parsing phenomena and that the above-mentioned reading time data can be
accounted for by a parallel-interactive model with conceptual and syntactic processors that do not pool their quality scores
together. Within the individual linguistic modules, decision-making can very well be competition-based.
51
Snijders, Tineke M.; Vosse, Theo; Kempen, Gerard; Van Berkum Jos J.A.; Petersson, Karl Magnus &
Hagoort, Peter (2009). Retrieval and unification of syntactic structure in sentence comprehension: an fMRI
study using word-category ambiguity. Cerebral Cortex, 19, 1493-1503.
Sentence comprehension requires the retrieval of single word information from long-term memory, and the integration of this
information into multiword representations. The current functional magnetic resonance imaging study explored the hypothesis
that the left posterior temporal gyrus supports the retrieval of lexicalsyntactic information, whereas left inferior frontal gyrus
(LIFG) contributes to syntactic unification. Twenty-eight subjects read sentences and word sequences containing word-category
(noun-verb) ambiguous words at critical positions. Regions contributing to the syntactic unification process should show
enhanced activation for sentences compared to words, and only within sentences display a larger signal for ambiguous than
unambiguous conditions. The posterior LIFG showed exactly this predicted pattern, confirming our hypothesis that LIFG
contributes to syntactic unification. The left posterior middle temporal gyrus was activated more for ambiguous than
unambiguous conditions (main effect over both sentences and word sequences), as predicted for regions subserving the retrieval
of lexical-syntactic information from memory. We conclude that understanding language involves the dynamic interplay between
left inferior frontal and left posterior temporal regions.
Harbusch, Karin & Kempen, Gerard (2002). A quantitative model of word order and movement in
English, Dutch, and German complement constructions. In Proceedings of the 19th International Conference
on Computational Linguistics (COLING-2002), Taipei (Taiwan). San Francisco: Morgan Kaufmann. [pp.
328-334]
We present a quantitative model of word order and movement constraints that enables a simple and uniform
treatment of a seemingly heterogeneous collection of linear order phenomena in English, Dutch and German
complement constructions (Wh-extraction, clause union, extraposition, verb clustering, particle movement, etc.).
Underlying the scheme are central assumptions of the psycholinguistically motivated Performance Grammar (PG).
Here we describe this formalism in declarative terms based on typed feature unification. PG allows a homogenous
treatment of both the within- and between-language variations of the ordering phenomena under discussion, which
reduce to different settings of a small number of quantitative parameters.
Kempen, Gerard; Olsthoorn, Nomi & Sprenger, Simone (in press, 2011). Grammatical workspace sharing
during language production and language comprehension: Evidence from grammatical multitasking.
Language and Cognitive Processes.
Grammatical encoding and grammatical decoding (in sentence production and comprehension, respectively) are
often portrayed as independent modalities of grammatical performance that only share declarative resources: lexicon
and grammar. The processing resources subserving these modalities are supposed to be distinct. In particular, one
assumes the existence of two workspaces where grammatical structures are assembled and temporarily maintained—
one for each modality. An alternative theory holds that the two modalities share many of their processing resources
and postulates a single mechanism for the online assemblage and short-term storage of grammatical structures: a
shared workspace. We report two experiments with a novel “grammatical multitasking” paradigm: The participants
had to read (i.e. decode) and to paraphrase (encode) sentences presented in fragments, responding to each input
fragment as fast as possible with a fragment of the paraphrase. The main finding was that grammatical constraints
with respect to upcoming input that emanate from decoded sentence fragments are immediately replaced by
grammatical expectations emanating from the structure of the corresponding paraphrase fragments. This evidences
52
that the two modalities have direct access to, and operate upon, the same (i.e. token-identical) grammatical
structures. This is possible only if the grammatical encoding and decoding processes command the same, shared
grammatical workspace. Theoretical implications for important forms of grammatical multitasking—self-monitoring,
turn-taking in dialogue, speech shadowing, and simultaneous translation—are explored.
Vosse, Theo & Kempen, Gerard (2008). Parsing verb-final clauses in German: Garden- path and ERP
effects modeled by a parallel dynamic parser. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings
of the 30th Annual Conference of the Cognitive Science Society (Washington DC, July 2008). Austin, TX:
Cognitive Science Society. [pp.261-266].
Experimental sentence comprehension studies have shown that superficially similar German clauses with verb-
final word order elicit very different garden-path and ERP effects. We show that a computer implementation of the
Unification Space parser (Vosse & Kempen, 2000) in the form of a localist-connectionist network can model the
observed differences, at least qualitatively. The model embodies a parallel dynamic parser that, in contrast with
existing models, does not distinguish between consecutive first-pass and reanalysis stages, and does not use
semantic or thematic roles. It does use structural frequency data and animacy information.
Lee
Arbib, M. A. and J. Lee (2007). Vision and Action in the Language-Ready Brain: From Mirror Neurons to
SemRep. BVAI 2007 (Brain Vision & Artificial Intelligence, 2007), LNCS 4729. F. Mele. Berlin/Heidelberg,
Springer-Verlag: 104-123.
The general setting for our work is to locate language perception and production within the broader context of
brain mechanisms for action and perception in general, modeling brain function in terms of the competition and
cooperation of schemas. Particular emphasis is placed on mirror neurons – neurons active both for execution of a
certain class of actions and for recognition of a (possibly broader) class of similar actions. We build on the early
VISIONS model of schema-based computer analysis of static scenes to present SemRep, a graphical representation
of dynamic visual scenes designed to support the generation of varied descriptions of episodes. Mechanisms for
parsing and production of sentences are currently being implemented within Template Construction Grammar
(TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics.
Arbib, M. A. and J. Lee (2008). Describing visual scenes: Towards a neurolinguistics based on construction
grammar. Brain Research 1225: 146-162.
The present paper is part of a larger effort to locate the production and perception of language within the broader
context of brain mechanisms for action and perception more generally. Here we model function in terms of the
competition and cooperation of schemas. We use the task of describing visual scenes to explore the suitability of
Construction Grammar as an appropriate framework for a schema-based linguistics. We recall the early VISIONS
model of schema-based computer analysis of static visual scenes and then introduce SemRep as a graphical
representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes. We
report preliminary results on implementing the production of sentences using Template Construction Grammar
(TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics. We
summarize data on neural correlates relevant to future work on TCG within the context of neurolinguistics, and
show how the relation between SemRep and TCG can serve as the basis for modeling language comprehension.
53
MacWhinney
MacWhinney, B. (2005). A Unified Model of Language Acquisition. In J. Kroll & A. De Groot (Eds.)
Handbook of bilingualism: Psycholinguistic approaches. Oxford University Press.
This paper presents an extended formulation of the Competition Model. The extended model is designed to account for a
larger range of phenomena in first and second language acquisition, including bilingualism. As in the classic version of the
Competition Model, competition is at the core of a set of non-modular interacting forces. However, now the various inputs to
competition are described in terms of six additional subcomponents: arenas, cues, chunking, storage, codes, and resonance.
Learning is viewed as a resonant process that relies on storage, chunking, and support to acquire new mappings.
Hernandez, A., Li, P., & MacWhinney, B. (2005). The emergence of competing modules in bilingualism.
Trends in Cognitive Sciences, 9, 220-225.
How does the brain manage to store and process multiple languages without encountering massive interference and transfer?
Unless we believe that bilinguals live in two totally unconnected cognitive worlds, we would expect far more transfer than
actually occurs. However, imaging and lesion studies have not provided consistent evidence for the strict neuronal separation
predicted by the theory of modularity. We suggest that emergentist theory offers a promising alternative. It emphasizes the
competitive interplay between multiple languages during childhood and by focusing on the dual action of competition and
entrenchment, avoids the need to invoke a critical period to account for age of acquisition effects in second-language learning.
This view instantiates the motto formulated by Elizabeth Bates that‘modules are made, not born.’
MacWhinnney, B. (2005). The emergence of linguistic form in time. Connection Science, 17, 191-211.
Linguistic forms are shaped by forces operating on vastly different time scales. Some of these forces operate directly at the
moment of speaking, whereas others accumulate over time in personal and social memory. Our challenge is to understand how
forces with very different time scales mesh together in the current moment to determine the emergence of linguistic form.
MacWhinney, B. (2008). How mental models encode embodied linguistic perspectives. In Klatzky, R.,
MacWhinney, B., and Behrmann, M. (Eds.). Embodiment, Ego-space, adn Action (pp. 365-405). Lawrence
Erlbaum.
Humans demonstrate a remarkable ability to take other people’s perspectives. When we watch movies, we find ourselves
identifying with the actors, sensing their joys, hopes, fears, and sorrows. As viewers, we can be moved to exhilaration as we
watch our heroes overcome obstacles; or we can be moved to tears when they suffer losses and defeats. This process of
identification does not always have to be linked to intense emotional involvement. At a soccer match, we can follow the
movements of a player moving in to shoot for a goal. We can identify with the player’s position, stance, and maneuvers against
the challenges offered by the defenders. We can track the actions, as the player drives toward the goal and kicks the ball into the
net. This ability to take the perspective of another person is very general. Just as we follow the movements of dancers, actors, and
athletes, we can also follow the thoughts and emotions expressed by others in language. In this paper, we will explore the ways in
which language builds upon our basic system for projecting the body image to support a rich system of perspective tracking and
mental model construction.
Li, P., Farkas, I., & MacWhinney, B. (2004). Early lexical development in a self-organizing neural
network. Neural Networks 17, 1345-1362.
In this paper we present a self-organizing neural network model of early lexical development called DevLex. The network
consists of two self-organizing maps (a growing semantic map and a growing phonological map) that are connected via
associative links trained by Hebbian learning. The model captures a number of important phenomena that occur in early lexical
acquisition by children, as it allows for the representation of a dynamically changing linguistic environment in language learning.
In our simulations, DevLex develops topographically organized representations for linguistic categories over time, models lexical
54
confusion as a function of word density and semantic similarity, and shows age-of-acquisition effects in the course of learning a
growing lexicon. These results match up with patterns from empirical research on lexical development, and have significant
implications for models of language acquisition based on self-organizing neural networks.
Li, P., Zhao, X., and MacWhinney, B. (2007) Dynamic Self-Organization and Early Lexical Development
in Children, Cognitive Science 31 581–612
In this study we present a self-organizing connectionist model of early lexical development.We call this model
DevLex-II, based on the earlier DevLex model. DevLex-II can simulate a variety of empirical patterns in children’s
acquisition of words. These include a clear vocabulary spurt, effects of word frequency and length on age of
acquisition, and individual differences as a function of phonological short-term memory and associative capacity.
Further results from lesioned models indicate developmental plasticity in the network’s recovery from damage, in a
non-monotonic fashion. We attribute the network’s abilities in accounting for lexical development to interactive
dynamics in the learning process. In particular, variations displayed by the model in the rate and size of early
vocabulary development are modulated by (a) input characteristics, such as word frequency and word length, (b)
consolidation of lexical-semantic representation, meaning-form association, and phonological short-term memory,
and (c) delayed processes due to interactions among timing, severity, and recoverability of lesion. Together, DevLex
and DevLex-II provide an accurate computational account of early lexical development.
MacWhinney, B. Item-based Patterns in Early Syntactic Development
This paper presents an account of first language acquisition based on the child�
s learning of item-based patterns
(IBPs). These patterns involve grammatical dependencies between a lexical predicate (such as more) and its
arguments (such as milk) to form a new cluster (such as more milk). Children can use simple, systematic inductive
operations to acquire these positional patterns, and then to generalize them into fuller feature-based and global
constructions. Together, these patterns can provide a full account of the learning of syntax. Recent work uses this
framework to construct computational simulations of children�
s syntactic development that match up well with
extensive, publicly available, corpus data available from the CHILDES database.
MacWhinney, B. The Logic of the Unified Model, In S. Gass and A. Mackey (Ed.) Handbook of Second
Language Acquisition – Routledge.
Many people believe that learning a second language is fundamentally different from learning a first language.
Evidence of this fundamental difference comes from the fact that first language acquisition almost invariably
produces full native speaker competence, whereas many second language learners achieve only partial success in
learning their new language. Some researchers believe that this difference in levels of ultimate attainment result
arises because, after the expiration of a certain critical period, the learning mechanisms that subserve first language
learning atrophy or expire. The Unified Competition Model (MacWhinney, 2008b) takes a different approach to this
issue. Instead of attributing differences between first and second language learning to the effects of a critical period,
these differences are attributed to the differential interplay between riskgenerating processes and protective, support
processes. For L1 learning, the five risk factors are entrenchment, parasitism, misconnection, negative transfer, and
isolation. To overcome these five risk factors, adults can rely on the support processes of resonance, internalization,
chunking, positive transfer, and participation. All of these risk factors and support processes are available to children,
as well as adults. What differs between L1 and L2 learning is the way in which these processes are configured.
55
Miikkulainen
Grasemann, U., Hoffman, R., and Miikkulainen, R. (in press). Modeling Acute and Compensated
Language Disturbance in Schizophrenia. In Proceedings of the 33rd Annual Meeting of the Cognitive Science
Society.
http:/nn.cs.utexas.edu/?grasemann:cogsci11
No current laboratory test can reliably identify patients with schizophrenia. Instead, key symptoms are observed
via language, including derailments, where patients cannot follow a coherent storyline, and delusions, where false
beliefs are repeated as fact. Brain processes underlying these and other symptoms remain unclear, and characterizing
them would greatly enhance our understanding of schizophrenia. In this situation, computational models can be
valuable tools to formulate testable hypotheses and to complement clinical research. This work aims to capture the
link between biology and schizophrenic symptoms using DISCERN, a connectionist model of human story
processing. Competing illness mechanisms proposed to underlie schizophrenia are simulated in DISCERN, and are
evaluated at the level of narrative language, i.e. the same level used to diagnose patients. The result is the first
simulation of abnormal storytelling in schizophrenia, both in acute psychotic and compensated stages of the
disorder. Of all illness models tested, hyperlearning, a model of overly intense memory consolidation, produced the
best fit to the language abnormalities of stable outpatients, as well as compelling models of acute psychotic
symptoms. If validated experimentally, the hyperlearning hypothesis could advance the current understanding of
schizophrenia, and provide a platform for developing future treatments for this disorder.
Hoffman, R.E., Grasemann, U., Gueorguieva, R., Quinlan, D., Lane, D., and Miikkulainen, R. (2011).
Using Computational Patients to Evaluate Illness Mechanisms in Schizophrenia, Biological Psychiatry.
http://nn.cs.utexas.edu/?grasemann:biopsych11
Background: Various malfunctions involving working memory, semantics, prediction error, and dopamine
neuromodulation have been hypothesized to cause disorganized speech and delusions in schizophrenia.
Computational models may provide insights into why some mechanisms are unlikely, suggest alternative
mechanisms, and tie together explanations of seemingly disparate symptoms and experimental findings.
Methods: Eight corresponding illness mechanisms were simulated in DISCERN, an artificial neural network
model of narrative understanding and recall. For this study, DISCERN learned sets of autobiographical and
impersonal crime stories with associated emotion coding. In addition, 20 healthy control subjects and 37 patients
with schizophrenia or schizoaffective disorder matched for age, gender, and parental education were studied using a
delayed story recall task. A goodness-of-fit analysis was performed to determine the mechanism best reproducing
narrative breakdown profiles generated by healthy control subjects and patients with schizophrenia. Evidence of
delusion-like narratives was sought in simulations best matching the narrative breakdown profile of patients.
Results: All mechanisms were equivalent in matching the narrative breakdown profile of healthy control subjects.
However, exaggerated prediction-error signaling during consolidation of episodic memories, termed hyperlearning,
was statistically superior to other mechanisms in matching the narrative breakdown profile of patients. These
simulations also systematically confused autobiographical agents with impersonal crime story agents to model fixed,
self-referential delusions.
56
Conclusions: Findings suggest that exaggerated prediction-error signaling in schizophrenia intermingles and
corrupts narrative memories when incorporated into long-term storage, thereby disrupting narrative language and
producing fixed delusional narratives. If further validated by clinical studies, these computational patients could
provide a platform for developing and testing novel treatments.
Miikkulainen, R., Kiran, S. (2009). Modeling the Bilingual Lexicon of an Individual Subject, In
Proceedings of the Workshop on Self-Organizing Maps (WSOM'09), Berlin, Springer Lexicon is a central component in any language processing system, whether human or artificial. Recent empirical evidence
suggests that a multilingual lexicon consists of a single component representing word meanings, and separate component for the
symbols in each language. These components can be modeled as self-organizing maps, with associative connections between
them implementing comprehension and production. Computational experiments in this paper show that such a model can trained
to match the proficiency and age of acquisition of particular bilingual individuals. In the future, it may be possible to use such
models to predict the effect of rehabilitation of bilingual aphasia, resulting in more effective treatments.
Grasemann, U., Sandberg, C., Kiran, S., and Miikkulainen, R. (2011). Impairment and Rehabilitation in
Bilingual Aphasia: A SOM-Based Model. In \Proceedings of the Eighth Workshop on Self-Organizing Maps
(WSOM'2011, Espoo, Finland).
http://nn.cs.utexas.edu/?grasemann:wsom11
ilingual aphasia is of increasing interest because a large and growing proportion of the world's population is
bilingual. Current clinical research on this topic cannot provide specific recommendations on which language
treatment should focus in a bilingual aphasic individual and to what extent cross-language transfer occurs during or
after rehabilitation. This paper describes a SOM-based model of the bilingual lexicon, and reports on simulations of
impairment and rehabilitation in bilingual aphasia. The goal is to create computational methods that can complement
clinical research in developing a better understanding of mechanisms underlying recovery, and that could be used in
the future to predict the most beneficial treatment for individual patients.
Williams, P., Miikkulainen, R. (2006). Grounding Language in Descriptions of Scenes, In Proceedings of
the 28th Annual Meeting of the Cognitive Science Society
The problem of how abstract symbols, such as those in systems of natural language, may be grounded in perceptual
information presents a significant challenge to several areas of research. This paper presents the GLIDES model, a neural
network architecture that shows how this symbol-grounding problem can be solved through learned relationships between simple
visual scenes and linguistic descriptions. Unlike previous models of symbol grounding, the model's learning is completely
unsupervised, utilizing the principles of self organization and Hebbian learning and allowing direct visualization of how concepts
are formed and grounding occurs. Two sets of experiments were conducted to evaluate the model. In the first set, linguistic test
stimuli were presented and the scenes that were generated by the model were evaluated as the grounding of the language. In the
second set, the model was presented with visual test samples and its language generation capabilities based on the grounded
representations were assessed. The results demonstrate that symbols can be grounded based on associations of perceptual and
linguistic representations, and the grounding can be made transparent. This transparency leads to unique insights into symbol
grounding, including how many-to-many mappings between symbols and referents can be maintained and how concepts can be
formed from cooccurrence relationships.
Nielsen
The Brede database: a small database for functional neuroimaging
http://www2.imm.dtu.dk/~fn/Nielsen2003Brede_abstract/Nielsen2003Brede_abstract.html
57
Abstract: Introduction -- We describe the Brede neuroinformatics database that provides data for novel
information retrieval techniques and automated meta-analyses. Data -- The database is inspired by the hierarchical
structure of BrainMap [1] with scientific articles (bib structures) on the highest level containing one or more
experiments (exp structure, corresponding to a contrast in gen- eral linear model analyses), these in turn comprising
one or more locations (loc structures). The informa- tion on the bib level (author, title, ...) is setup automatically
from PubMed while the rest of the informa- tion is entered manually in a Matlab graphical user interface. On the loc
level this includes the 3D stereotac- tic coordinates in either Talairach or MNI space, the brain area (functional,
anatomical or cytoarchitectonic area) and magnitude values such as Z-score and P-value. On the exp level
information such as modality, scanner and behavioral domain are recorded with external components (such as face
recognition or kinetic bound- aries) organized in a directed graph and marked up with Medical Subject Headings
(MeSH) where possible. Items in the database are identified with unique numbers and the type of identifier is given
a unique string, e.g., WOBIB: 27 for an Epstein and Kanwisher paper. This will allow Internet search engine to
identify the phrase. For storing the data we employ a simple XML format that we denote poor-man’s XML (pXML)
with no attributes and no empty tags. The database presently con- sists of data constructed from 40 scientific
articles, containing 134 experiments and 882 locations...
Mining for associations between text and brain activation in a functional neuroimaging database
http://www.springerlink.com/content/m01l058504070750/
Abstract: We describe a method for mining a neuroimaging database for associations between text and brain
locations. The objective is to discover association rules between words indicative of cognitive function as described
in abstracts of neuroscience papers and sets of reported stereotactic Talairach coordinates. We invoke a simple
probabilistic framework in which kernel density estimates are used to model distributions of brain activation foci
conditioned on words in a given abstract. The principal associations are found in the joint probability density
between words and voxels. We show that the statistically motivated associations are well aligned with general
neuroscientific knowledge.
Modeling of activation data in the BrainMapTM database: Detection of outliers
www2.imm.dtu.dk/~fn/ps/Nielsen2000Modeling_text.ps
Abstract: We describe a system for meta-analytical modeling of activation foci from functional neuroimaging
studies. Our main vehicle is a set of density models in Talairach space capturing the distribution of activation foci in
sets of experiments labeled by lobar anatomy. One important use of such density models is identification of novelty,
i.e., low probability database events. We rank the novelty of the outliers and investigate the cause for 21 of the most
novel, finding several outliers that are entry and transcription errors or infrequent or non-conforming terminology.
We briefly discuss the use of atlases for outlier detection.
Lost in localization: A solution with neuroinformatics 2.0?
http://onlinelibrary.wiley.com/doi/10.1002/hbm.10012/abstract
Abstract: The commentary by Derrfuss and Mar (Derrfuss, J., Mar, R.A., 2009. Lost in localization: The need for
a universal coordinate database. NeuroImage, doi:10.1016/j.neuroimage.2009.01.053.) discusses some of the
limitations of the present databases and calls for a universal coordinate database. Here I discuss further issues and
propose another angle to the solution of a universal coordinate database with the use of wiki technology.
58
Visualizing data mining results with the Brede tools
http://www.frontiersin.org/neuroinformatics/10.3389/neuro.11/026.2009/abstract
A few neuroinformatics databases now exist that record results from neuroimaging studies in the form of brain
coordinates in stereotaxic space. The Brede Toolbox was originally developed to extract, analyze and visualize data
from one of them the BrainMap database. Since then the Brede Toolbox has expanded and now includes its own
database with coordinates along with ontologies for brain regions and functions: The Brede Database. With Brede
Toolbox and Database combined, we setup automated workflows for extraction of data, mass meta-analytic data
mining and visualizations. Most of the Web presence of the Brede Database is established by a single script
executing a workflow involving these steps together with a final generation of Web pages with embedded
visualizations and links to interactive three-dimensional models in the Virtual Reality Modeling Language. Apart
from the Brede tools I briefly review alternate visualization tools and methods for Internet-based visualization and
information visualization as well as portals for visualization tools.
Oztop
All can be downloaded from http://www.cns.atr.jp/~erhan/ErhanOztop_publications.html:
Chaminade T, Oztop E, Cheng G, Kawato M. From self-observation to imitation: visuomotor association
on a robotic hand. Brain Res Bull. 2008 Apr 15;75(6):775-84. Epub 2008 Feb 14.
Being at the crux of human cognition and behaviour, imitation has become the target of investigations ranging
from experimental psychology and neurophysiology to computational sciences and robotics. It is often assumed that
the imitation is innate, but it has more recently been argued, both theoretically and experimentally, that basic forms
of imitation could emerge as a result of self-observation. Here, we tested this proposal on a realistic experimental
platform, comprising an associative network linking a 16 degrees of freedom robotic hand and a simple visual
system. We report that this minimal visuomotor association is sufficient to bootstrap basic imitation. Our results
indicate that crucial features of human imitation, such as generalization to new actions, may emerge from a
connectionist associative network. Therefore, we suggest that a behaviour as complex as imitation could be, at the
neuronal level, founded on basic mechanisms of associative learning, a notion supported by a recent proposal on the
developmental origin of mirror neurons. Our approach can be applied to the development of realistic cognitive
architectures for humanoid robots as well as to shed new light on the cognitive processes at play in early human
cognitive development.
Oztop, E, Imamizu, H, Cheng, G, Kawato, M (2006) A computation model of anterior intraparietal (AIP)
neurons. Neurocomputing, 69(10-12): 1354-1361.
The monkey parietal anterior intraparietal area (AIP) is part of the grasp planning and execution circuit which
contains neurons that encode object features relevant for grasping, such as the width and the height. In this study we
focus on the formation of AIP neurons during grasp development. We propose and implement a neural network
structure and a learning mechanism that is driven by successful grasp experiences during early grasp development.
The simulations show that learning leads to emergence of units that have similar response properties as the AIP
visual-dominant neurons. The results may have certain implications for the function of AIP neurons and thus should
stimulate new experiments that cannot only verify/falsify the model but also advance our understanding of the
visuomotor learning mechanisms employed by the primate brain.
59
Oztop E, Kawato M, Arbib M (2006) Mirror Neurons and Imitation: A Computationally Guided Review.
Neural Networks 19: 254-271
Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals
the presence of ‘mirror systems’ (not individual neurons) in the human. Current conceptual models attribute high
level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of
these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on
mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of
current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror
systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular
selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often
overlook the computational requirements for posited functions, while too many computational models adopt the
erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines
the gap between conceptual and computational models and points out the research effort required from both sides to
reduce this gap. q 2006 Published by Elsevier Ltd. Keywords: Mirror neuron; Action understanding; Imitation;
Language; Computational model
Oztop E, Wolpert D, Kawato M (2005) Mental state inference using visual control parameters. Cogn Brain
Res 22: 129-151
Although we can often infer the mental states of others by observing their actions, there are currently no
computational models of this remarkable ability. Here we develop a computational model of mental state inference
that builds upon a generic visuomanual feedback controller, and implements mental simulation and mental state
inference functions using circuitry that subserves sensorimotor control. Our goal is (1) to show that control
mechanisms developed for manual manipulation are readily endowed with visual and predictive processing
capabilities and thus allows a natural extension to the understanding of movements performed by others; and (2) to
give an explanation on how cortical regions, in particular the parietal and premotor cortices, may be involved in
such dual mechanism. To analyze the model, we simulate tasks in which an observer watches an actor performing
either a reaching or a grasping movement. The observer’s goal is to estimate the dmental stateT of the actor: the goal
of the reaching movement or the intention of the agent performing the grasping movement. We show that the motor
modules of the observer can be used in a dsimulation modeT to infer the mental state of the actor. The simulations
with different grasping and non-straight line reaching strategies show that the mental state inference model is
applicable to complex movements. Moreover, we simulate deceptive reaching, where an actor imposes false beliefs
about his own mental state on an observer. The simulations show that computational elements developed for
sensorimotor control are effective in inferring the mental states of others. The parallels between the model and
cortical organization of movement suggest that primates might have developed a similar resource utilization strategy
for action understanding, and thus lead to testable predictions about the brain mechanisms of mental state inference.
Oztop E., Arbib M.A. (2002) Schema Design and Implementation of the Grasp-Related Mirror Neuron
System. Biological Cybernetics 87: (2) 116-140
Mirror neurons within a monkey’s premotor area F5 fire not only when the monkey performs a certain class of
actions but also when the monkey observes another monkey (or the experimenter) perform a similar action. It has
60
thus been argued that these neurons are crucial for understanding of actions by others. We o�er the hand-state
hypothesis as a new explanation of the evolution of this capability: the basic functionality of the F5 mirror system is
to elaborate the appropriate feedback – what we call the hand state – for opposition-space based control of manual
grasping of an object. Given this functionality, the social role of the F5 mirror system in understanding the actions of
others may be seen as an exaptation gained by generalizing from one’s own hand to an other’s hand. In other words,
mirror neurons first evolvedto augment the ‘‘canonical’’ F5 neurons (active during self-movement based on
observation of an object) by providing visual feedback on ‘‘handstate,’’ relating the shape of the hand to the shape
of the object. We then introduce the MNS1 (mirror neuron system 1) model of F5 and related brain regions. The
existing Fagg–Arbib–Rizzolatti–Sakata model represents circuitry for visually guided grasping of objects, linking
the anterior intraparietal area (AIP) with F5 canonical neurons. The MNS1 model extends the AIP visual pathway
by also modeling pathways, directed toward F5 mirror neurons, which match arm– hand trajectories to the
a�ordances and location of a potential target object. We present the basic schemas for the MNS1 model, then
aggregate them into three ‘‘grand schemas’’ – visual analysis of hand state, reach and grasp, andthe core mirror
circuit – for each of which we present a useful implementation (a non-neural visual processing system, a multijoint
3-D kinematics simulator, and a learning neural network, respectively). With this implementation we show how the
mirror system may learn to recognize actions already in the repertoire of the F5 canonical neurons. We show that the
connectivity pattern of mirror neuron circuitry can be estab- lishedthrough training, andthat the resultant network
can exhibit a range of novel, physiologically interesting behaviors during the process of action recognition. We train
the system on the basis of final grasp but then observe the whole time course of mirror neuron activity, yielding
predictions for neurophysiological experiments under conditions of spatial perturbation, altered kinematics,
andambiguous grasp execution which highlight the importance of the timing of mirror neuron activity.
Ugur E, Oztop E, Sahin E (2011) Goal emulation and planning in perceptual space using learned
affordances. Robotics and Autonomous Systems 59, 580-595
In this paper, we show that through self-interaction and self-observation, an anthropomorphic robot equipped
with a range camera can learn object affordances and use this knowledge for planning. In the first step of learning,
the robot discovers commonalities in its action-effect experiences by discovering effect categories. Once the effect
categories are discovered, in the second step, affordance predictors for each behavior are obtained by learning the
mapping from the object features to the effect categories. After learning, the robot can make plans to achieve desired
goals, emulate end states of demonstrated actions, monitor the plan execution and take corrective actions using the
perceptual structures employed or discovered during learning. We argue that the learning system proposed shares
crucial elements with the development of infants of 7–10 months age, who explore the environment and learn the
dynamics of the objects through goal-free exploration. In addition, we discuss goal emulation and planning in
relation to older infants with no symbolic inference capability and non-linguistic animals which utilize object
affordances to make action plans.
Schilling
Schilling, M., Cruse, H.: Cognition as recruitment of reactive systems. (submitted,
http://www.icsi.berkeley.edu/~lucag/archive/manuscript_schilling_cruse%20Reactive%20Based%20Cognition%20
AfterSubm%20Layouted.pdf ).
61
It has often been stated that a neuronal system to become a cognitive one has to be complex enough, complexity
often paralleled with size. In contrast, we argue that basic requirements can be fulfilled already by small neuronal
systems raising the question what are the crucial neuronal structures and processes that make a system a cognitive
one. Our main claim is that it is not possible to understand cognition without taking into account the reactive basis of
an embedded cognitive system. Therefore, we propose a network, termed reaCog, that, first, is able to deal with a
specific domain of behavior (six-legged-walking). Second, we show how this system can be expanded to be able to
plan ahead and invent behaviors in order to solve new problems. The expansion consists of a manipulable body
model and a WTA network that governs the activation of the behaviors used for planning ahead
Schilling, M. (2011): Universally manipulable body models—dual quaternion representations in layered
and dynamic MMCs. Autonomous Robots 30:4, 399–425; http://dx.doi.org/10.1007/s10514-011-9226-3
Surprisingly complex tasks can be solved using a behaviour-based, reactive control system, i.e., a system that
operates without an explicit internal representation of the environment and the own body. Nevertheless, application
of internal representations has gained interest in recent years because such internal representations can be used to
solve problems of perception and motor control (sensor fusion, inverse modeling) and may in addition be applied to
higher cognitive functions as are the ability to plan ahead. To endow such a system with the ability to find new
behavioural solutions to a given problem in a broad range of possibilities, the internal representation must be
universally manipulable, i.e. the model should be able to simulate all movements that are physically possible for the
body given. Using recurrent neural networks, models showing this faculty have been proposed being based on the
principle of mean of multiple computation (MMC). The extension of this approach to three dimensions requires the
introduction of a joint angle representation which allows for computation of mean values. Here we use dual
quaternions that are singularity-free and unambiguous which allow for shortest path interpolation. In addition, it has
been shown that dual quaternions are the most efficient and most compact form for representing rigid
transformations. The model can easily be adapted to bodies of arbitrary geometries. The extended MMC net
introduced in this article represents a holistic system that can—following the principle of pattern completion—
likewise be used as an inverse model, a forward model, for sensor fusion or other, related capabilities.
Schilling, M. (in press): Learning by seeing—associative learning of visual features through mental
simulation of observed action. ECAL 2011, Paris; http://www.icsi.berkeley.edu/~mschilli/schilling2011ecal.pdf
Internal representations employed in cognitive tasks have to be embodied. The flexible use of such grounded
models allows for higher-level function like planning ahead, cooperation and communication. But at the same time
this flexibility presupposes that the utilized internal models are interrelating multiple modalities. In this article we
present how an internal body model serving motor control tasks can be recruited for learning to recognize
movements performed by another agent. We show that—as the movements are governed by an equal underlying
internal model—it is sufficient to observe the other agent performing a series of movements and that there is no
supervised learning necessary, i.e. the learning agent does not require access to the performing agents postural
information (joint configurations). Instead, through the shared underlying dynamics the mapping can be
bootstrapped by the observing agent from the sequence of visual input features.
Small
Small et al. (2011) - From Language Comprehension to Action Understanding and Back Again
62
A controversial question in cognitive neuroscience is whether comprehension of words and sentences engages
brain mechanisms specific for decoding linguistic meaning or whether language comprehension occurs through
more domain-general sensorimotor processes. Accumulating behavioral and neuroimaging evidence suggests a role
for cortical motor and premotor areas in passive action-related language tasks, regions that are known to be involved
in action execution and observation. To examine the involvement of these brain regions in language and
nonlanguage tasks, we used functional magnetic resonance imaging (fMRI) on a group of 21 healthy adults. During
the fMRI session, all participants 1) watched short object-related action movies, 2) looked at pictures of man-made
objects, and 3) listened to and produced short sentences describing object-related actions and man-made objects. Our
results are among the first to reveal, in the human brain, a functional specialization within the ventral premotor
cortex (PMv) for observing actions and for observing objects, and a different organization for processing sentences
describing actions and objects. These findings argue against the strongest version of the simulation theory for the
processing of action-related language.
Small et al. (2011) - Interpretation-mediated changes in neural activity during language comprehension
Using functional magnetic resonance imaging (fMRI), we identified cortical regions mediating interpretive
processes that take place during language comprehension. We manipulated participants' interpretation of texts by
asking them to focus on action-, space-, or time-related features while listening to identical short stories. We identify
several cortical regions where activity varied significantly in response to this attention manipulation, even though
the content being processed was exactly the same. Activity in the posterior and anterior sections of the left inferior
frontal gyrus (IFG), which are thought to have different sensitivities to high-level language processing, was
modulated by the listeners' attentional focus, but in ways that were quite different. The posterior left IFG (Pars
Opercularis) showed different activity levels for the three conditions. However, a population coding analysis
demonstrated similar distributions of activity across conditions. This suggests that while the gain of the response in
the Pars Opercularis was modulated, its core organization was relatively invariant across the experimental
conditions. In the anterior left IFG (Pars Triangularis), the analysis of population codes revealed different activity
patterns between conditions: there was little similarity between activity during time-attention and action- and space-
attention, however there were similar activity patterns while attending to space and action information. In addition,
both the left superior temporal gyrus and sulcus showed greater activity in the space and action attention conditions
when contrasted with time attention. We discuss these findings in light of work on the role of left IFG in processing
semantic information in language, and in light of theories suggesting that temporal information in language is
processed in the brain using similar mechanisms as spatial information. Our findings suggest that a substantial
source of variance in neural activity during language comprehension emerges from the internally-driven,
information-seeking preferences of listeners rather than the syntactic or semantic properties of a text.
Small et al. (2009) - Gestures Orchestrate Brain Networks for Language Understanding
Although the linguistic structure of speech provides valuable communicative information, nonverbal behaviors
can offer additional, often disambiguating cues. In particular, being able to see the face and hand movements of a
speaker facilitates language comprehension [1]. But how does the brain derive meaningful information from these
movements? Mouth movements provide information about phonological aspects of speech [2-3]. In contrast,
cospeech gestures display semantic information relevant to the intended message [4-6]. We show that when
63
language comprehension is accompanied by observable face movements, there is strong functional connectivity
between areas of cortex involved in motor planning and production and posterior areas thought to mediate
phonological aspects of speech perception. In contrast, language comprehension accompanied by cospeech gestures
is associated with tuning of and strong functional connectivity between motor planning and production areas and
anterior areas thought to mediate semantic aspects of language comprehension. These areas are not tuned to hand
and arm movements that are not meaningful. Results suggest that when gestures accompany speech, the motor
system works with language comprehension areas to determine the meaning of those gestures. Results also suggest
that the cortical networks underlying language
Vanduffel
Orban GA, Van Essen D, Vanduffel W. Comparative mapping of higher visual areas in monkeys and
humans. Trends Cogn Sci. 2004 Jul;8(7):315-24.
The advent of functional magnetic resonance imaging (fMRI) in non-human primates has facilitated comparison
of the neurobiology of cognitive functions in humans and macaque monkeys, the most intensively studied animal
model for higher brain functions. Most of these comparative studies have been performed in the visual system. The
early visual areas V1, V2 and V3, as well as the motion area MT are conserved in humans. Beyond these areas,
differences between human and monkey functional organization are increasingly evident. At the regional level, the
monkey inferotemporal and intraparietal complexes appear to be conserved in humans, but there are profound
functional differences in the intraparietal cortex suggesting that not all its constituent areas are homologous. In the
long term, fMRI offers opportunities to compare the functional anatomy of a variety of cognitive functions in the
two species.
Nelissen K, Borra E, Gerbella M, Rozzi S, Luppino G, Vanduffel W, Rizzolatti G, Orban GA. Action
observation circuits in the macaque monkey cortex. J Neurosci. 2011 Mar 9;31(10):3743-56.
In both monkeys and humans, the observation of actions performed by others activates cortical motor areas. An
unresolved question concerns the pathways through which motor areas receive visual information describing motor
acts. Using functional magnetic resonance imaging (fMRI), we mapped the macaque brain regions activated during
the observation of grasping actions, focusing on the superior temporal sulcus region (STS) and the posterior parietal
lobe. Monkeys viewed either videos with only the grasping hand visible or videos with the whole actor visible.
Observation of both types of grasping videos activated elongated regions in the depths of both lower and upper
banks of STS, as well as parietal areas PFG and anterior intraparietal (AIP). The correlation of fMRI data with
connectional data showed that visual action information, encoded in the STS, is forwarded to ventral premotor
cortex (F5) along two distinct functional routes. One route connects the upper bank of the STS with area PFG, which
projects, in turn, to the premotor area F5c. The other connects the anterior part of the lower bank of the STS with
premotor areas F5a/p via AIP. Whereas the first functional route emphasizes the agent and may relay visual
information to the parieto-frontal mirror circuit involved in understanding the agent's intentions, the second route
emphasizes the object of the action and may aid in understanding motor acts with respect to their immediate goal.
Vanduffel W, Fize D, Peuskens H, Denys K, Sunaert S, Todd JT, Orban GA. Extracting 3D from motion:
differences in human and monkey intraparietal cortex. Science. 2002 Oct 11;298(5592):413-5.
64
We compared three-dimensional structure-from-motion (3D-SFM) processing in awake monkeys and humans
using functional magnetic resonance imaging. Occipital and midlevel extrastriate visual areas showed similar
activation by 3D-SFM stimuli in both species. In contrast, intraparietal areas showed significant 3D-SFM activation
in humans but not in monkeys. This suggests that human intraparietal cortex contains visuospatial processing areas
that are not present in monkeys.
Wood
Action comprehension in non-human primates: motor simulation or inferential reasoning?
Wood & Hauser (2008) Trends in Cognitive Sciences Vol.12 No.12, 461-465.
Some argue that action comprehension is intimately connected with the observer’s own motor capacities,
whereas others argue that action comprehension depends on non-motor inferential mechanisms. We address this
debate by reviewing comparative studies that license four conclusions: monkeys and apes extract the meaning of an
action (i) by going beyond the surface properties of actions, attributing goals and intentions to the agent; (ii) by
using environmental information to infer when actions are rational; (iii) by making predictions about an agent’s
goal, and the most probable action to obtain the goal given environmental constraints; (iv) in situations in which
they are physiologically incapable of producing the actions. Motor theories are, thus, insufficient to account for
primate action comprehension in the absence of inferential mechanisms.
The Perception of Rational, Goal-Directed Action in Nonhuman Primates
Wood, Glynn, Phillips & Hauser Science 317, 1402 (2007)
Humans are capable of making inferences about other individuals' intentions and goals by evaluating their
actions in relation to the constraints imposed by the environment. This capacity enables humans to go beyond the
surface appearance of behavior to draw inferences about an individual's mental states. Presently unclear is whether
this capacity is uniquely human or is shared with other animals. We show that cotton-top tamarins, rhesus macaques,
and chimpanzees all make spontaneous inferences about a human experimenter's goal by attending to the
environmental constraints that guide rational action. These findings rule out simple associative accounts of action
perception and show that our capacity to infer rational, goal-directed action likely arose at least as far back as the
New World monkeys, some 40 million years ago.
The uniquely human capacity to throw evolved from a non-throwing primate: an evolutionary dissociation
between action and perception
Wood, Glynn & Hauser (2007) Biology Letters
Humans are uniquely endowed with the ability to engage in accurate, high-momentum throwing. Underlying this
ability is a unique morphological adaptation that enables the characteristic rotation of the arm and pelvis. What is
unknown is whether the psychological mechanisms that accompany the act of throwing are also uniquely human.
Here we explore this problem by asking whether free-ranging rhesus monkeys (Macaca mulatta), which lack both
the morphological and neural structures to throw, nonetheless recognize the functional properties of throwing.
Rhesus not only understand that human throwing represents a threat, but that some aspects of a throwing event are
more relevant than others; specifically, rhesus are sensitive to the kinematics, direction and speed of the rotating
arm, the direction of the thrower's eye gaze and the object thrown. These results suggest that the capacity to throw
did not coevolve with psychological mechanisms that accompany throwing; rather, this capacity may have built
65
upon pre-existing perceptual processes. These results are consistent with a growing body of work showing that non-
human animals often exhibit perceptual competencies that do not show up in their motor responses, suggesting
evolutionary dissociations between the systems of perception that provide understanding of the world and those that
mediate action on the world.