TAC 2012 Cold Start...5/13/13! 1! TAC 2012" Cold Start" Knowledge Base Population! When Lisa's...

5/13/13

1

TAC 2012��Cold Start��

Knowledge Base Population

When Lisa's mother Marge Simpson went to a weekend getaway at Rancho Relaxo, the movie The Happy Little Elves Meet Fuzzy Snuggleduck was one of the R-rated european adult movies available on their cable channels.

After two years in the academic quagmire of Springfield Elementary, Lisa finally has a teacher that she connects with. But she soon learns that the problem with being middle-class is that

Homer Simpson

Bart Simpson Lisa Simpson

Marge Simpson

Springfield Elementary

Springfield



Homer Simpson


Marge Simpson


Springfield

Bottomless Pete, Nature’s Cruelest Mistake

per:children per:children

per:alternate_names per:cities_of_residence

per:spouse

per:schools_attended



5/13/13

2

Cold Start

Schema

Entity-Valued Relations

Relation Inverse(s) per:children per:parentsper:other_family per:other_familyper:parents per:childrenper:siblings per:siblingsper:spouse per:spouseper:employee_of {org,gpe}:employees*per:member_of org:membership*per:schools_attended org:students*per:city_of_birth gpe:births_in_city*per:stateorprovince_of_birth gpe:births_in_stateorprovince*per:country_of_birth gpe:births_in_country*per:cities_of_residence gpe:residents_of_city*per:statesorprovinces_of_residence gpe:residents_of_stateorprovinceper:countries_of_residence gpe:residents_of_country*per:city_of_death gpe:deaths_in_city*per:stateorprovince_of_death gpe:deaths_in_stateorprovince*per:country_of_death gpe:deaths_in_country*org:shareholders {per,org,gpe}:holds_shares_in*org:founded_by {per,org,gpe}:organizations_founded*org:top_members_employees per:top_member_employee_of*{org,gpe}:member_of org:membersorg:members {org,gpe}:member_oforg:parents {org,gpe}:subsidiariesorg:subsidiaries org:parentsorg:city_of_headquarters gpe:headquarters_in_city*org:stateorprovince_of_headquarters gpe:headquarters_in_stateorprovince*org:country_of_headquarters gpe:headquarters_in_country*

String-Filled Relations per:alternate_names org:alternate_names

per:date_of_birth org:political_religious_affiliation

per:age org:number_of_employees_members

per:origin org:date_founded

per:date_of_death org:date_dissolved

per:cause_of_death org:website

per:title

per:religion

per:charges

Cold Start

Schema per:childrenper:other_familyper:parentsper:siblingsper:spouseper:employee_ofper:member_ofper:schools_attendedper:city_of_birthper:stateorprovince_of_birthper:country_of_birthper:cities_of_residenceper:statesorprovinces_of_residenceper:countries_of_residenceper:city_of_deathper:stateorprovince_of_deathper:country_of_deathorg:shareholdersorg:founded_by

5/13/13

3

The Task











But wait, there’s more!!

You are given:

The Task











Lisa Marge Simpson Rancho Relaxo

Fuzzy Snuggleduck

You are given:

Homer Simpson


Marge Simpson


Springfield




per:spouse












You Must Produce:

Homer Simpson


Marge Simpson


Springfield




per:spouse


How do you know that your KB is any good?

5/13/13

4

Homer Simpson


Marge Simpson


Springfield




per:spouse


How do you know that your KB is any good?

Align it to a ground truth KB ☞ How do you know

that your KB is any good? Align it to a ground

truth KB ☞

But how are you going to produce ground truth? And wouldn’t the

alignment be intractable anyway if the KB were of any reasonable

size?

Homer Simpson


Marge Simpson


Springfield




per:spouse


Where did the children of Marge Simpson go to school?

per:children per:schools_attended

Homer Simpson


Marge Simpson


Springfield




per:spouse




5/13/13

5

Query Entity First Relation Second Relation Adriana Petryna per:title

Blackstone Group org:founded_by

William Shore per:organizations_founded org:date_founded

Wistar Institute org:employees per:title

Andrew W. Mellon per:children per:organizations_founded

Lycee Alliance Israelite Universelle org:employees per:schools_attended

Tsitsi Jaji per:schools_attended org:students

Sample Evaluation Queries

6 November 2012

KBP-09

Pipeline

serif /facets

.tac

.xml

Validator.pl .tac.

err

COE.predicates.txt

.tac

s2t

.tac1

.fac1 facetsfille

r

.fac_out

altnames

.altnames

.tac2

jackknife –r –c –p

.tac3

topempl rmerrors .tac4 ValidateC

oldStart.pl

.tac5 .tac3a

cullassertions

.tac6 jackknife –m –p rmerrors

.tac8 RequireInverses.pl

.tac9 .tac7

ValidateColdStart.p

l

.tac10

timex2 .tac11

Freshly minted KB

Corpus

CALE

CUNY

5/13/13

6

6 November 2012

KBP-09

Practice: Washington Post articles

26k 2010 Washington Post articles (194k assertions) 6 November 2012

KBP-09

Demo: Query Engine

l  Since no training data was available, we assessed our practice KBs qualitatively

26k 2010 Washington Post articles (194k assertions)

6 November 2012

KBP-09

Good KELVIN

l  Jared Fogle is an employee of Subway l  Freeman Hrabowski works for UMBC, founded the

Meyerhoff Scholors Program, and attended both Hampton University and the University of Illinois

l  Elena Kagan attended Oxford, Harvard, and Princeton

l  The Applied Physics Laboratory is a subsidiary of Johns Hopkins University

l  Southwest Airlines is headquartered in Texas

26k 2010 Washington Post articles (194k assertions) 6 November 2012

KBP-09

Bad KELVIN

l  Harry Reid is an employee of the Republican Party. (KELVIN also learns he is employee of the Democratic Party) Ø  Tend to see more trouble with popular entities

l  Big Foot is an employee of Starbucks l  Steven Spielberg lives in Iran l  Jill Biden is married to Jill Biden

26k 2010 Washington Post articles (194k assertions)

5/13/13

7

6 November 2012

KBP-09

1 year of WP = 1 computer scientist

l  KELVIN learns: Ø  ("ian soboroff”) per:employee_of ("national institute of

standards and technology") Ø  ("ian soboroff") per:title "computer scientist”

l  paulmac@hpcc1: more WPB_ENG_20100506.0012.sgm l  One scholar Anderson has been talking with is Dan Cohen, the

director of the Center for History and New Media at George Mason University; another is Ian Soboroff, a computer scientist with the National Institute of Standards and Technology, who studies how well technology works with human language. Soboroff hopes that in the future, Twitter can be used to help researchers understand how information was dispersed and what human networks in the early 21st century looked like.

6 November 2012

KBP-09

Experimental Results

CUNY CALE P R F1 P R F1

hltcoe1 0.5256 0.2843 0.3690 0.1620 0.2182 0.1859

hltcoe2 YES 0.4929 0.3031 0.3753 0.1818 0.2406 0.2071

hltcoe3 YES 0.4865 0.5000 0.4932 0.1849 0.3510 0.2423

hltcoe4 YES YES 0.4799 0.5155 0.4971 0.1842 0.4168 0.2555

hltcoe5 YES 0.4937 0.3053 0.3773 0.1453 0.2531 0.1846

0-hop slots 1-hop slots

CALE (better entity linking) materially improves recall

Precision (unsurprisingly) is lower for 0-hop vs. 1-hop

6 November 2012

KBP-09

Jena RDF database

Fuseki RDF triple store

query service

Pubby RDF

linked data

Browser

Text search

interface

6 November 2012

KBP-09

Some reasoning usecases

l Deriving new relations and properties husband_of(X,Y) => male(X)

l Default reasoning religion(X,R) ^ member(X,C) ^ church(C) ^ unknown(affiliation(C,?)) => affiliation(C,R)

l Detecting inconsistencies birthDate(X,D1) ^ deathDate(X,D2) ^ before(D2,D1)

l Detecting unusual or anomalous data A person with more than 10 children

l Providing evidence for/against entity merging

TAC 2012 Cold Start...5/13/13! 1! TAC 2012" Cold Start" Knowledge Base Population! When Lisa's...

Documents

Transcript of TAC 2012 Cold Start...5/13/13! 1! TAC 2012" Cold Start" Knowledge Base Population! When Lisa's...