CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN Massimo Poesio Uni Trento,...

CONCEPTUAL KNOWLEDGE: CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, EVIDENCE FROM CORPORA, THE MIND, AND THE BRAINTHE MIND, AND THE BRAIN

Massimo PoesioMassimo PoesioUni Trento, Center for Mind / Brain SciencesUni Trento, Center for Mind / Brain Sciences

Uni Essex, Language & ComputationUni Essex, Language & Computation

(joint work with A. Almuhareb, (joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy)E. Barbu, M. Baroni, B. Murphy)

MOTIVATIONSMOTIVATIONS

Research on conceptual knowledge is Research on conceptual knowledge is carried out in Computational Linguistics, carried out in Computational Linguistics, Neural Science, and PsychologyNeural Science, and Psychology

But there is limited interchange between But there is limited interchange between CL and the other disciplines studying CL and the other disciplines studying concepts concepts – Except indirectly through the use of WordNetExcept indirectly through the use of WordNet

This work: use data from Psychology and This work: use data from Psychology and Neural Science to evaluate (vector-space) Neural Science to evaluate (vector-space) models produced in CLmodels produced in CL

OUTLINEOUTLINE

Vector space representations Vector space representations A `semantic’ vector space modelA `semantic’ vector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature

normsnorms Category distinctions and Brain dataCategory distinctions and Brain data

CONCEPTUAL SEMANTICS IN CONCEPTUAL SEMANTICS IN VECTOR SPACEVECTOR SPACE

LEXICAL ACQUISITION IN LEXICAL ACQUISITION IN CORPUS / COMP LINGCORPUS / COMP LING

Vectorial representations of lexical Vectorial representations of lexical meaning derived from IRmeaning derived from IR

WORD-BASED vector models: WORD-BASED vector models: – vector dimensions are words vector dimensions are words – Schuetze 91, 98; HAL, LSA, Turney, Schuetze 91, 98; HAL, LSA, Turney,

RappRapp GRAMMATICAL RELATION models:GRAMMATICAL RELATION models:

– vector dimensions are pairs <Rel,W>vector dimensions are pairs <Rel,W>– Grefenstette 93, Lin 98, Curran&Moens, Grefenstette 93, Lin 98, Curran&Moens,

Pantel, Widdows, Pado & Lapata, …..Pantel, Widdows, Pado & Lapata, …..

FEATURES IN VECTOR SPACE FEATURES IN VECTOR SPACE MODELSMODELS

GRAMMATICAL RELATIONS

WORDS

STRENGHTS OF THIS STRENGHTS OF THIS APPROACH: CATEGORIZATIONAPPROACH: CATEGORIZATION

LIMITATIONS OF THIS WORKLIMITATIONS OF THIS WORK

Very simplistic view of concepts Very simplistic view of concepts – In fact, typically extract lexical In fact, typically extract lexical

representations for WORDS (non-representations for WORDS (non-disambiguated)disambiguated)

Limited evaluationLimited evaluation– Typical evaluation: judges’ opinions Typical evaluation: judges’ opinions

about correctness of distances / about correctness of distances / comparing with WordNetcomparing with WordNet

Most work not connected with work Most work not connected with work on concepts in Psychology / Neural on concepts in Psychology / Neural ScienceScience

OUR WORKOUR WORK

Acquire richer, more semantic-Acquire richer, more semantic-oriented concept descriptions by oriented concept descriptions by exploiting relation extraction exploiting relation extraction techniquestechniques

Develop task-based methods for Develop task-based methods for evaluating the resultsevaluating the results

Integrate results from corpora with Integrate results from corpora with results from psychology & neural results from psychology & neural sciencescience

THIS TALKTHIS TALK

Acquire richer, more semantic-Acquire richer, more semantic-oriented concept descriptions by oriented concept descriptions by exploiting relation extraction exploiting relation extraction techniquestechniques

Develop task-based methods for Develop task-based methods for evaluating the resultsevaluating the results

Integrate results from Integrate results from corpora with results from corpora with results from psychology & neural sciencepsychology & neural science

OUTLINEOUTLINE

Vector space representations Vector space representations A `semantic’ vector space modelA `semantic’ vector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature


MORE ADVANCED THEORIES MORE ADVANCED THEORIES OF CONCEPTSOF CONCEPTS

In Linguistics:In Linguistics:– Pustejovsky Pustejovsky

In AI:In AI:– Description LogicsDescription Logics– Formal ontologiesFormal ontologies

In Psychology:In Psychology:– Theory Theory (Murphy, 2002)Theory Theory (Murphy, 2002)– FUSS (Vigliocco Vinson et al)FUSS (Vigliocco Vinson et al)

SEMANTIC CONCEPT DESCRIPTIONSSEMANTIC CONCEPT DESCRIPTIONSPUSTEJOVSKY (1991, 1995)PUSTEJOVSKY (1991, 1995)

Lexical entries have a QUALIA STRUCTURE Lexical entries have a QUALIA STRUCTURE consisting of four ‘roles’consisting of four ‘roles’– FORMAL role: what type of object it is (shape, FORMAL role: what type of object it is (shape,

color, ….)color, ….)– CONSTITUTIVE role: what it consists of (parts, CONSTITUTIVE role: what it consists of (parts,

stuff, etc.)stuff, etc.) E.g., for books, chapters, index, paper ….E.g., for books, chapters, index, paper ….

– TELIC role: what is the purpose of the object TELIC role: what is the purpose of the object (e.g., for books, READING)(e.g., for books, READING)

– AGENTIVE role: how the object was created AGENTIVE role: how the object was created (e.g., for books, WRITING)(e.g., for books, WRITING)

BEYOND BUNDLES OF ATTRIBUTES: BEYOND BUNDLES OF ATTRIBUTES: DESCRIPTION LOGICS, THEORY THEORYDESCRIPTION LOGICS, THEORY THEORY

We know much more about concepts than We know much more about concepts than the fact that they have certain attributes:the fact that they have certain attributes:– We know that cars have 4 wheels whereas We know that cars have 4 wheels whereas

bicycles have 2bicycles have 2– We don’t just know that people have heads, We don’t just know that people have heads,

bodies and legs, but that heads are attached in bodies and legs, but that heads are attached in certain positions whereas legs are attached in certain positions whereas legs are attached in other onesother ones

– Facts of this type can be expressed even in the Facts of this type can be expressed even in the simplest concept description languages, those simplest concept description languages, those of description logicsof description logics

BEYOND SIMPLE RELATIONS: BEYOND SIMPLE RELATIONS: DESCRIPTION LOGICSDESCRIPTION LOGICS

Bear (and Animal ( 4 Paw) …)

Strawberry (and Fruit (fills Color red) … )

Female (and Human ( Male))

WORD SENSE WORD SENSE DISCRIMINATIONDISCRIMINATION

The senses of The senses of palmpalm in WordNet in WordNet1.1. the inner surface of the hand from the the inner surface of the hand from the

wrist to the base of the fingerswrist to the base of the fingers2.2. a linear unit based on the length or a linear unit based on the length or

width of the human handwidth of the human hand3.3. any plant of the family Palmae having any plant of the family Palmae having

an unbranched trunk crowned by large an unbranched trunk crowned by large pinnate or palmate leavespinnate or palmate leaves

4.4. an award for winning a championship an award for winning a championship or commemorating some other eventor commemorating some other event

CONCEPT ACQUISITION MEETS CONCEPT ACQUISITION MEETS RELATION EXTRACTIONRELATION EXTRACTION

– We developed methods to identify We developed methods to identify SEMANTICSEMANTIC properties of concepts properties of concepts (`Deep lexical relations’)(`Deep lexical relations’)ATTRIBUTES and their VALUES ATTRIBUTES and their VALUES

– Almuhareb & Poesio 2004, 2005Almuhareb & Poesio 2004, 2005Extracting QUALIA Extracting QUALIA

– Poesio & Almuhareb 2005Poesio & Almuhareb 2005Letting relations emerge from the data: Letting relations emerge from the data:

STRUDELSTRUDEL– Baroni et al, Cognitive Science to appearBaroni et al, Cognitive Science to appear

Extracting Wu & Barsalou –style relationsExtracting Wu & Barsalou –style relations– Poesio Barbu Giuliano & Romano, 2008Poesio Barbu Giuliano & Romano, 2008

We showed that for a variety of tasks such We showed that for a variety of tasks such conceptual descriptions are ‘better’ than conceptual descriptions are ‘better’ than word-based or grammatical function-based word-based or grammatical function-based descriptionsdescriptions

ALMUHAREB & POESIO 2005: ALMUHAREB & POESIO 2005: USING A PARSERUSING A PARSER

LOOKING ONLY FOR (POTENTIAL) ATTRIBUTES AND THEIR VALUES BETTER THAN USING ALL GRS

EVEN IF ATTRIBUTES OBTAINED USING TEXT PATTERNS (“THE X OF THE Y” )

ATTRIBUTES AND VALUES VS. ATTRIBUTES AND VALUES VS. ALL CORPUS FEATURESALL CORPUS FEATURES

DescriptionDescription Vector sizeVector size

500500 15221522 30443044 47534753 49694969

Values onlyValues only 64.86%64.86% 94.59%94.59% -- - - 94.59%94.59%

Attributes onlyAttributes only 97.30%97.30% 97.30%97.30% 97.3097.30%%

--

Attributes (1522) Attributes (1522) & Values (1522)& Values (1522)

-- -- 100%100% -- --

SUPERVISED EXTRACTION OF SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS CONCEPT DESCRIPTIONS

Using a theory of attributes merging Using a theory of attributes merging ideas from Pustejovsky and Guarino ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)(Poesio and Almuhareb, 2005)

Using Wu and Barsalou’s theory of Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & attributes (Poesio Barbu Romano & Giuliano, 2008)Giuliano, 2008)

THE CLASSIFICATION SCHEME FOR THE CLASSIFICATION SCHEME FOR ATTRIBUTES OF POESIO & ALMUHAREB ATTRIBUTES OF POESIO & ALMUHAREB

20052005 PART PART

– (cfr. Guarino’s non-relational attributes, Pustejovsky’s (cfr. Guarino’s non-relational attributes, Pustejovsky’s constitutive roles)constitutive roles)

RELATED OBJECT RELATED OBJECT – Non-relational attributes other than parts, relational Non-relational attributes other than parts, relational

rolesroles QUALITYQUALITY

– Guarino’s qualities, Pustejovsky’s formal rolesGuarino’s qualities, Pustejovsky’s formal roles ACTIVITY ACTIVITY

– Pustejosvky’s telic and agentive rolesPustejosvky’s telic and agentive roles RELATED AGENTRELATED AGENT NOT AN ATTRIBUTE (= everything elseNOT AN ATTRIBUTE (= everything else))

A SUPERVISED FEATURE A SUPERVISED FEATURE CLASSIFIERCLASSIFIER

We developed a supervised feature We developed a supervised feature classifier that relies on 4 types of classifier that relies on 4 types of informationinformation– Morphological info (Dixon, 1991)Morphological info (Dixon, 1991)– Question patternsQuestion patterns– Features of featuresFeatures of features– Feature useFeature use

Some nouns used more commonly as features than Some nouns used more commonly as features than as concepts: i.e., “the F of the C is” more frequent as concepts: i.e., “the F of the C is” more frequent than “the * of the F is”than “the * of the F is”

(These last four methods all rely on info (These last four methods all rely on info extracted from the Web)extracted from the Web)

THE EXPERIMENTTHE EXPERIMENT We created a BALANCED DATASETWe created a BALANCED DATASET

– ~ ~ 400400 concepts concepts – representing representing all 21 WordNetall 21 WordNet classes, including both classes, including both

ABSTRACT and CONCRETE conceptsABSTRACT and CONCRETE concepts– balanced as to balanced as to ambiguityambiguity and and frequencyfrequency

We collected from the Web We collected from the Web 20,00020,000 candidate candidate features of these concepts using patternsfeatures of these concepts using patterns

We hand-classified We hand-classified 1,1551,155 candidate features candidate features We used these data to train We used these data to train

– A binary classifier (feature / non feature)A binary classifier (feature / non feature)– A 5-way classifierA 5-way classifier

OUTLINEOUTLINE

Vector space representations Vector space representations An example of `Semantic-based’ An example of `Semantic-based’

vector space modelvector space modelEvaluating such modelsEvaluating such models Attribute extraction and Feature Attribute extraction and Feature


EVALUATIONEVALUATION

– Qualitative:Qualitative:Visual inspectionVisual inspectionAsk subjects to assess correctness of the Ask subjects to assess correctness of the

classification of the attributesclassification of the attributes

– Quantitative:Quantitative:Use conceptual descriptions for CLUSTERING Use conceptual descriptions for CLUSTERING

(CATEGORIZATION)(CATEGORIZATION)

VISUAL EVALUATION: VISUAL EVALUATION: TOP 400 FEATURES OF DEER TOP 400 FEATURES OF DEER

ACCORDING TO OUR CLASSIFIERACCORDING TO OUR CLASSIFIER

ClassClass AttributesAttributes

Parts & Parts & Related Related ObjectsObjects

antlers, leg, carcass, head, eyes, skin, body, blood, track, neck, antlers, leg, carcass, head, eyes, skin, body, blood, track, neck, horns, flesh, meat, legs, hide, loin, chest, throat, tongue, heart, horns, flesh, meat, legs, hide, loin, chest, throat, tongue, heart, horn, coat, trail, tail, bones, ears, scent, home, coverts, nose, horn, coat, trail, tail, bones, ears, scent, home, coverts, nose, feet, shoulder, stomach, foot, sight, rack, skull, hair, intestines, feet, shoulder, stomach, foot, sight, rack, skull, hair, intestines, necks, line, brain, belly, tendons, step, heads, mind, entrails, necks, line, brain, belly, tendons, step, heads, mind, entrails, skins, hooves, cells, bell, cavity, skins, hooves, cells, bell, cavity, picturepicture, testicles, , testicles, photophoto, , forehead, genitals, knees, innards, rump, butt, fur, face, shank, forehead, genitals, knees, innards, rump, butt, fur, face, shank, brains, brains, imageimage, ear, , ear, statuestatue, path, corpse, jaw, bladder, muzzle, , path, corpse, jaw, bladder, muzzle, calf, hoofs, abdomen, calf, hoofs, abdomen, hillhill, quarters, shit, senses, paths, wound, , quarters, shit, senses, paths, wound, stream, feces, varieties, protector, hoof, stream, feces, varieties, protector, hoof, picspics, nostrils, portrait, , nostrils, portrait, liver, flanks, pen, forest, vitals, side, hips, garden, food, muscle, liver, flanks, pen, forest, vitals, side, hips, garden, food, muscle, muscles, guts, droppings, bodies, veil, footprints, wounds, muscles, guts, droppings, bodies, veil, footprints, wounds, hearts, homes, teeth, underside, breast, turn, haunches, hearts, homes, teeth, underside, breast, turn, haunches, forelegs, brow, hip, figure, torso, village, spring, chin, tails, forelegs, brow, hip, figure, torso, village, spring, chin, tails, organs, enclosure, lips, hindquarters, valley, cave, flank, figures, organs, enclosure, lips, hindquarters, valley, cave, flank, figures, tissues, spot, insides, dick, backs, skeleton, barktissues, spot, insides, dick, backs, skeleton, bark

VISUAL EVALUATION: VISUAL EVALUATION: QUALITIESQUALITIES

ClassClass AttributesAttributes

QualityQuality

deathdeath, age, beauty, sex, form, sense, reaction, cry, terror, curiosity, , age, beauty, sex, form, sense, reaction, cry, terror, curiosity, health, fleetness, appetite, health, fleetness, appetite, survivalsurvival, condition, thirst, life, ways, , condition, thirst, life, ways, swiftness, plight, whereabouts, swiftness, plight, whereabouts, fatefate, grace, gender, , grace, gender, sicknesssickness, , need, perspective, need, perspective, slaughterslaughter, , capturecapture, modesty, ecology, , modesty, ecology, preservationpreservation, detriment, agility, heartbeat, greed, gentleness, , detriment, agility, heartbeat, greed, gentleness, behaviour, behavior, aggressiveness, screams, favor, predicament, behaviour, behavior, aggressiveness, screams, favor, predicament, genetics, honours, elegance, propensity, reactions, harvest, genetics, honours, elegance, propensity, reactions, harvest, rescuerescue, curse, mercy, gaze, sustainability, intelligence, lives, , curse, mercy, gaze, sustainability, intelligence, lives, thoughtsthoughts

QUANTITATIVE EVALUATIONQUANTITATIVE EVALUATION ATTRIBUTES ATTRIBUTES

– PROBLEM: can’t compare against WordNetPROBLEM: can’t compare against WordNet– Precision / recall against hand-annotated Precision / recall against hand-annotated

datasetsdatasets– Human judges (ourselves):Human judges (ourselves):

We used the classifiers to classify the top 20 features We used the classifiers to classify the top 20 features of 21 randomly chosen conceptsof 21 randomly chosen concepts

We separately evaluated the resultsWe separately evaluated the results

CATEGORIES:CATEGORIES:– Clustering of the balanced datasetClustering of the balanced dataset– PROBLEM: PROBLEM: The WordNet category structure is The WordNet category structure is

highly subjectivehighly subjective

ATTRIBUTE CLASSIFICATIONATTRIBUTE CLASSIFICATION

DescriptionDescription Cross-ValidationCross-Validation Human Judge Human Judge (AA)(AA)

Human Judge Human Judge (MP)(MP)

Correctly Correctly ClassifiedClassified

Instances Instances (Accuracy)(Accuracy)

928 (out of 1155)928 (out of 1155)80.35%80.35%

244 (out of 365)244 (out of 365)66.85%66.85%

182 (out of 260)182 (out of 260)70.00%70.00%

MeasureMeasure PP RR FF PP RR FF PP RR FF

ActivityActivity 0.8220.822 0.8780.878 0.8490.849 0.8330.833 0.5930.593 0.6930.693 0.8750.875 0.6220.622 0.7270.727

Part/Related-Part/Related-ObjectObject

0.8420.842 0.8820.882 0.8620.862 0.8320.832 0.7120.712 0.7670.767 0.7930.793 0.8020.802 0.7980.798

QualityQuality 0.7990.799 0.8210.821 0.8100.810 0.7670.767 0.6290.629 0.6910.691 0.8640.864 0.6460.646 0.7390.739

Related-AgentRelated-Agent 0.9300.930 0.9700.970 0.9500.950 0.8150.815 0.9170.917 0.8630.863 0.8260.826 0.8260.826 0.8260.826

Not-AttributeNot-Attribute 0.6020.602 0.4870.487 0.5380.538 0.3390.339 0.6270.627 0.4400.440 0.2970.297 0.6330.633 0.4040.404

CLUSTERING WITH 2-WAY CLUSTERING WITH 2-WAY CLASSIFIERCLASSIFIER

AllAllCandidate Candidate

AttributesAttributesHeuristic filteringHeuristic filtering Filtering by Filtering by

classificationclassification

PurityPurity 0.6570.657 0.6720.672 0.6930.693

EntropyEntropy 0.3350.335 0.319 0.319 0.302 0.302

Vector sizeVector size 24,17824,178 4,2964,296 3,8243,824

Clustered Clustered ConceptsConcepts 402402 402402 401401

CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSIS

ANIMAL bear, bull, camel, cat, cow, deer, dog, elephant, horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle

CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSIS

ANIMAL bear, bull, camel, cat, cow, deer, dog, elephant, horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle

EDIBLE FRUIT

apple, banana, berry, cherry, fig, grape, kiwi, lemon, lime, mango, melon, olive, orange, peach, pear, pineapple, strawberry, watermelon, (pistachio, oyster)

CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSISANIMAL bear, bull, camel, cat, cow, deer, dog, elephant,

horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle

EDIBLE FRUIT


ILLNESS acne, anthrax, arthritis, asthma, cancer, cholera, cirrhosis, diabetes, eczema, flu, glaucoma, hepatitis, leukemia, malnutrition, meningitis, plague, rheumatism, smallpox, (superego, lumbago, neuralgia, sciatica, gestation, menopause, quaternary, pain)

CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSISANIMAL bear, bull, camel, cat, cow, deer, dog, elephant,

horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle

EDIBLE FRUIT


ILLNESS acne, anthrax, arthritis, asthma, cancer, cholera, cirrhosis, diabetes, eczema, flu, glaucoma, hepatitis, leukemia, malnutrition, meningitis, plague, rheumatism, smallpox, (superego, lumbago, neuralgia, sciatica, gestation, menopause, quaternary, pain)

IN WORDNET: PAIN

LIMITS OF THIS TYPE OF LIMITS OF THIS TYPE OF EVALUATIONEVALUATION

No way of telling how complete / No way of telling how complete / accurate are our concept accurate are our concept descriptionsdescriptions– Both in terms of relations and in terms Both in terms of relations and in terms

of their relative importance of their relative importance No way of telling whether the No way of telling whether the

category distinctions we get from category distinctions we get from WordNet are empirically foundedWordNet are empirically founded

BEYOND JUDGES / EVALUATION BEYOND JUDGES / EVALUATION AGAINST WORDNETAGAINST WORDNET

Task-based evaluationTask-based evaluation

Evidence from other areas of Evidence from other areas of cognitive sciencecognitive science

(ESSLLI 2008 Workshop - Baroni / (ESSLLI 2008 Workshop - Baroni / Evert / Lenci )Evert / Lenci )

TASK-BASED (BLACK-BOX) TASK-BASED (BLACK-BOX) EVALUATIONEVALUATION

Tasks requiring lexical knowledge:Tasks requiring lexical knowledge:– Lexical tests:Lexical tests:

TOEFL test (Rapp 2001, Turney 2005)TOEFL test (Rapp 2001, Turney 2005)

– NLP tasks:NLP tasks:Eg, anaphora resolution (Poesio et al 2004)Eg, anaphora resolution (Poesio et al 2004)

– Actual applicationsActual applicationsE.g., language models (Mitchell & Lapata E.g., language models (Mitchell & Lapata

ACL 2009, Lapata invited talk)ACL 2009, Lapata invited talk)

EVIDENCE FROM OTHER AREAS EVIDENCE FROM OTHER AREAS OF COGNITIVE SCIENCEOF COGNITIVE SCIENCE

Attributes: evidence from psychologyAttributes: evidence from psychology– Association lists (priming)Association lists (priming)

E.g., use results of association tests to evaluate E.g., use results of association tests to evaluate proximity (Lund et al, 1995; Pado and Lapata, 2008)proximity (Lund et al, 1995; Pado and Lapata, 2008)

Comparison against feature norms: Schulte im Walde, Comparison against feature norms: Schulte im Walde, 2008)2008)

– Feature normsFeature norms

Category distinctions: evidence from Category distinctions: evidence from neural scienceneural science

OUTLINEOUTLINE


vector space modelvector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature


FEATURE-BASED REPRESENTATIONS IN FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGYPSYCHOLOGY

Feature-based concept representations assumed Feature-based concept representations assumed by many cognitive psychology theories (Smith by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997)and Medin, 1981, McRae et al, 1997)

Underpin development of prototype theory (Rosch Underpin development of prototype theory (Rosch et al)et al)

Used, e.g., to account for semantic priming Used, e.g., to account for semantic priming (McRae et al, 1997; Plaut, 1995)(McRae et al, 1997; Plaut, 1995)

Underlie much work on category-specific defects Underlie much work on category-specific defects (Warrington and Shallice, 1984; Caramazza and (Warrington and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004)Vigliocco, 2004)

FEATURE NORMSFEATURE NORMS

Subjects produce lists of features for Subjects produce lists of features for a concepta concept

Weighed by number of subjects that Weighed by number of subjects that produce themproduce them

Several existing (Rosch and Mervis, Several existing (Rosch and Mervis, Garrard et al, McRae et al, Vinson Garrard et al, McRae et al, Vinson and Vigliocco)and Vigliocco)

Substantial differences in collection Substantial differences in collection methodology and resultsmethodology and results

SPEAKER-GENERATED FEATURES SPEAKER-GENERATED FEATURES (VINSON AND VIGLIOCCO)(VINSON AND VIGLIOCCO)

COMPARING CORPUS FEATURES COMPARING CORPUS FEATURES WITH FEATURE NORMS WITH FEATURE NORMS

(Almuhareb et al 2005, Poesio et al 2007)(Almuhareb et al 2005, Poesio et al 2007)

35 concepts in common between the Almuhareb & 35 concepts in common between the Almuhareb & Poesio dataset and the dataset produced by Poesio dataset and the dataset produced by Vinson and Vigliocco (2002, 2003)Vinson and Vigliocco (2002, 2003)– ANIMALS: bear, camel, cat, cow, dog, elephant, horse, ANIMALS: bear, camel, cat, cow, dog, elephant, horse,

lion, mouse, sheep, tiger, zebralion, mouse, sheep, tiger, zebra– FRUIT: apple, banana, cherry, grape, lemon, orange, FRUIT: apple, banana, cherry, grape, lemon, orange,

peach, pear, pineapple, strawberry, watermelonpeach, pear, pineapple, strawberry, watermelon– VEHICLE: airplane, bicycle, boat, car, helicopter, VEHICLE: airplane, bicycle, boat, car, helicopter,

motorcycle, ship, truck, vanmotorcycle, ship, truck, van

We compared the features we obtained for these We compared the features we obtained for these concepts with the speaker-generated features concepts with the speaker-generated features collected by Vinson and Viglioccocollected by Vinson and Vigliocco

RESULTSRESULTS Best recall: ~ 52% (using all attributes and Best recall: ~ 52% (using all attributes and

values)values) Best precision: ~ 19%Best precision: ~ 19% But: high correlation But: high correlation (ro=.777) (ro=.777) between between

the distances between concept the distances between concept representations obtained from corpora, representations obtained from corpora, and the distances between the and the distances between the representations for the same concepts representations for the same concepts obtained from subjects (using the cosine obtained from subjects (using the cosine as a measure of similarity)as a measure of similarity)

DISCUSSIONDISCUSSION

– Substantial differences in features and Substantial differences in features and overlap, but correlation similaroverlap, but correlation similar

– Problems:Problems:Each feature norm slightly differentEach feature norm slightly differentThey have been normalized by hand: LOUD, They have been normalized by hand: LOUD,

NOISY, NOISE all mapped to LOUDNOISY, NOISE all mapped to LOUD

AN EXAMPLE: STRAWBERRYAN EXAMPLE: STRAWBERRY

Speaker-generated features:

Matching Features Collected Using Our Text Patterns – (with

frequency)

red (20) red (5), colour (5), color (1)

fruit (18) fruit (5)

sweet (13) sweetness (8)

has seeds (12) seeds (6), seed (2)

grows (10) growth (1), ripening (10)

small (6) size (19)

taste (6) taste (6), flavor (6), flavour (2)

food (5) nutrition (1)

from garden (5) cultivation (7), harvest (6), harvester (2)

juice (5) juice (10), juices (3)

dessert (3) sweetness (8)

eat (3) nutrition (1)

Problem: differences between Problem: differences between feature normsfeature norms

motorcyclemotorcycle– Vinson & Vigliocco:Vinson & Vigliocco:

wheel, motor, loud, vehicle, wheel, fast, handle, ride, wheel, motor, loud, vehicle, wheel, fast, handle, ride, transport, bike, human, danger, noise, seat, brake, drive, transport, bike, human, danger, noise, seat, brake, drive, fun, gas, machine, object, open, small, travel, windfun, gas, machine, object, open, small, travel, wind

– Garrard et al:Garrard et al: vehicle, wheel, fast, handlebar, light, seat, make a noise, vehicle, wheel, fast, handlebar, light, seat, make a noise,

tank, metal, unstable, tyre, coloured, sidecar, indicator, tank, metal, unstable, tyre, coloured, sidecar, indicator, pannier, pedal, speedometer, manoeuvrable, race, brakes, pannier, pedal, speedometer, manoeuvrable, race, brakes, stop, move, engine, petrol, economical, gearsstop, move, engine, petrol, economical, gears

– McRae et al:McRae et al: wheels, 2_wheels, dangerous, engine, fast, helmets, wheels, 2_wheels, dangerous, engine, fast, helmets,

Harley_Davidson, loud, 1_or_2_people, vehicle, leather, Harley_Davidson, loud, 1_or_2_people, vehicle, leather, transportation, 2_people, fun, Hell's_Angels, gasolinetransportation, 2_people, fun, Hell's_Angels, gasoline

– Mutual correlation of ranks ranges from 0.4 to 0.7Mutual correlation of ranks ranges from 0.4 to 0.7

DISCUSSIONDISCUSSION

– Preliminary conclusions: need to collect new Preliminary conclusions: need to collect new feature norms for CLfeature norms for CL

E.g., use similar techniques to collect attributes for E.g., use similar techniques to collect attributes for WordNetWordNet

See Kremer & Baroni 2008See Kremer & Baroni 2008

– For more work on using feature norms for For more work on using feature norms for conceptual acquisition, see conceptual acquisition, see

Schulte im Walde 2008Schulte im Walde 2008 Baroni et al to appearBaroni et al to appear

– For the correlation between feature norms and For the correlation between feature norms and information in WordNet (meronymy, isa, plus information in WordNet (meronymy, isa, plus info from glosses): Barbu & Poesio GWC 2008info from glosses): Barbu & Poesio GWC 2008

OUTLINEOUTLINE


vector space modelvector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature

normsnormsCategory distinctions and brain Category distinctions and brain

datadata

USING BRAIN DATA TO IDENTIFY USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONSCATEGORY DISTINCTIONS

Studies of brain-damaged patients have Studies of brain-damaged patients have been shown to provide useful insights in been shown to provide useful insights in the organization of conceptual knowledge the organization of conceptual knowledge in the brainin the brain– Warrington and Shallice 1984, Caramazza & Warrington and Shallice 1984, Caramazza &

Shilton 1998Shilton 1998 fMRI has been used to identify these fMRI has been used to identify these

distinctions in healthy patients as welldistinctions in healthy patients as well– E.g., Martin & ChaoE.g., Martin & Chao

See, e.g., Capitani et al 2003 for a surveySee, e.g., Capitani et al 2003 for a survey

CATEGORY DISTINCTIONS IN CATEGORY DISTINCTIONS IN THE BRAINTHE BRAIN

ANIMALSTOOLS

CORPUS DATA AND BRAIN CORPUS DATA AND BRAIN DATADATA

Can brain data (from healthy Can brain data (from healthy patients) be used to get an patients) be used to get an ‘objective’ picture of categorical ‘objective’ picture of categorical distinctions in the brain?distinctions in the brain?

Can our findings be useful to Can our findings be useful to understand better the neurological understand better the neurological results?results?

Ongoing project: using EEG and fMRI Ongoing project: using EEG and fMRI to identify such distinctionsto identify such distinctions

EEG Spectral Analysis of EEG Spectral Analysis of ConceptsConcepts

Participants presented Participants presented with aural or visual with aural or visual concept stimuli concept stimuli

EEG apparatus EEG apparatus records electrical records electrical activity on the scalpactivity on the scalp

Waveforms can be Waveforms can be reduced to frequency reduced to frequency componentscomponents

EEG vs. fMRIEEG vs. fMRI

EEG pros and consEEG pros and cons

Pros:Pros:– Lighter Lighter – CheaperCheaper– Better temporal resolution (ms)Better temporal resolution (ms)

Cons:Cons:– Coarser spatial resolution (cm)Coarser spatial resolution (cm)– Noisy (e.g., very sensitive to skull depth)Noisy (e.g., very sensitive to skull depth)

A CATEGORY DISTINCTION A CATEGORY DISTINCTION EXPERIMENT WITH EEGEXPERIMENT WITH EEG

Murphy Poesio Bovolo DalPonte & Murphy Poesio Bovolo DalPonte & Bruzzone, Cogsci 2008Bruzzone, Cogsci 2008

Seven Italian native speakersSeven Italian native speakers Image stimuli only:Image stimuli only:

– 30 tools30 tools– 30 animals30 animals

Each stimulus presented six timesEach stimulus presented six times Optimal time / frequency window identified Optimal time / frequency window identified

automatically automatically – 100-370ms100-370ms– 3-17 Hz3-17 Hz

Stimuli: Images from WebStimuli: Images from Web

Data analysisData analysis

PreprocessingPreprocessing– Artefact removalArtefact removal

Feature extraction Feature extraction – CSSD: a form of supervised component CSSD: a form of supervised component

analysisanalysis ClassificationClassification

– Using SVMsUsing SVMs

EEG SIGNALS: EEG SIGNALS: TIME-FREQUENCY (PER CHANNEL)TIME-FREQUENCY (PER CHANNEL)

Extraction of features (EEGOXELS) Extraction of features (EEGOXELS) from EEG datafrom EEG data

Data analysisData analysis

Classification System Schematic

Filte

r by T

ime

, F

req

an

d E

ele

ctr.

CS

SD

D

eco

mp

ositio

n

Ve

ctor T

ran

sform

Su

pV

ec M

ach

ine

var(“to

ol”),

var(“a

nim

al”)

64 channels preprocessed data

X channels filtered data* “Tool” component

“Animal” component

Feature vector

Answer

?

RESULTSRESULTS

Representation of categories in Representation of categories in CSSD spacesCSSD spaces

component analysis identifies 2-component analysis identifies 2-dimensional spacesdimensional spaces

Analysis of these spaces may provide Analysis of these spaces may provide useful data against which to compare useful data against which to compare our corpus modelsour corpus models

CSSD-derived conceptual spacesCSSD-derived conceptual spaces

BRAIN DATA AND CORPUS BRAIN DATA AND CORPUS DATADATA

What is the relation between the What is the relation between the conceptual spaces induced from conceptual spaces induced from corpora with the conceptual spaces corpora with the conceptual spaces elicited using EEG? elicited using EEG?

PREDICTING BRAIN (FMRI) ACTIVATION PREDICTING BRAIN (FMRI) ACTIVATION USING CONCEPT DESCRIPTIONSUSING CONCEPT DESCRIPTIONS

T. Mitchell, S. Shinkareva, A. Carlson, K. Chang, V. Malave, R. Mason and M. Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195

MITCHELL ET AL 2008: MITCHELL ET AL 2008: METHODSMETHODS

Record fMRI activation for 60 nominal conceptsRecord fMRI activation for 60 nominal concepts– And extract 200 ‘best’ features, or VOXELsAnd extract 200 ‘best’ features, or VOXELs

Build conceptual descriptions for these concepts Build conceptual descriptions for these concepts from corpora (the Web)from corpora (the Web)– 25 features for each concept 25 features for each concept – 25 verbs expressing typical properties of living things / 25 verbs expressing typical properties of living things /

toolstools– Collect strength of association between these features Collect strength of association between these features

and each conceptand each concept Learn association between each voxel and the 25 Learn association between each voxel and the 25

verbal features using 58 conceptsverbal features using 58 concepts Use learned model to predict activation of 2 held-Use learned model to predict activation of 2 held-

out data (compare using Euclidean distance)out data (compare using Euclidean distance)– Accuracy: 77%Accuracy: 77%

MITCHELL ET AL 2008MITCHELL ET AL 2008

MITCHELL ET AL 2008: MITCHELL ET AL 2008: VERB FEATURESVERB FEATURES

MITCHELL ET AL: MITCHELL ET AL: LEARNING ASSOCIATIONSLEARNING ASSOCIATIONS

OUR EXPERIMENTS OUR EXPERIMENTS Replicate the Mitchell et al study using Replicate the Mitchell et al study using

EEG data instead of fMRIEEG data instead of fMRI– Different feature selection mechanismsDifferent feature selection mechanisms

Compare different methods for building Compare different methods for building concept descriptionsconcept descriptions– In addition to hand-picked, also a variety of In addition to hand-picked, also a variety of

standard corpus modelsstandard corpus models For ItalianFor Italian B. Murphy, M. Baroni, M. Poesio, B. Murphy, M. Baroni, M. Poesio, EEG

responds to conceptual stimuli and corpus semantics, EMNLP 2009

RESULTS USING THE HAND-RESULTS USING THE HAND-PICKED FEATURESPICKED FEATURES

RESULTS USING AUTOMATICALLY RESULTS USING AUTOMATICALLY SELECTED FEATURESSELECTED FEATURES

MITCHELL ET AL

AA-MP

RECAPRECAP

We need to relate the evidence from We need to relate the evidence from corpora with evidence about concepts corpora with evidence about concepts coming from empirical work in the coming from empirical work in the neuroscience and psychologyneuroscience and psychology

Feature norms databases could be used Feature norms databases could be used to evaluate attribute extractionto evaluate attribute extraction– But: need to find better ways of collecting But: need to find better ways of collecting

themthem Brain data may give us information about Brain data may give us information about

the ‘real’ conceptual categoriesthe ‘real’ conceptual categories– Results still preliminary Results still preliminary

COLLABORATORSCOLLABORATORS

ABDULRAHMAN ALMUHAREB(Essex PhD 2006 Now at KACST, Saudi Arabia)

HEBA LAKANY(formerly Essex, now Strathclyde)

BRIAN MURPHY(Trento)

EDUARD BARBU(Trento PhD forthc)

MARCO BARONI(Trento)

THANKSTHANKS

To the audience …..

And to Galja, Ruslan & the other organizers for yet another splendid RANLP!

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN Massimo Poesio Uni Trento,...

Documents

Transcript of CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN Massimo Poesio Uni Trento,...