Linguistic Resources for the 2012 TAC KBP Slot Filling Evaluations

Linguistic Resources for the 2012 TAC KBP Slot Filling Evaluations

Joe Ellis (presenter), Brendan Callahan, Jonathan Wright, Stephanie Strassel

Linguistic Data ConsortiumUniversity of Pennsylvania, USA

Outline

English and Spanish source data Annotator and assessor guidelinesLabeled training and evaluation data

Annotation Tasks and MethodologiesEntity SelectionSlot Filling AnnotationSlot Filling Assessment

Linguistic Resources for 2012 Slot Filling

TAC KBP Evaluation Workshop – NIST, November 5-6, 2012

Source Corpora – 2012


Language Genre DocumentsBroadcast Conversation 17Broadcast News 665Conversation Telephone Speech 1

2,286,866

Web Text (2008 – 2009) 1,490,595

Spanish Newswire (2007 – 2010) 1,000,020

EnglishNewswire (2007 – 2010)

Language Source Nodes Catalog IDEnglish 2008 Wikipedia 818, 741 LDC2009E58

Source Corpora

SF Annotator/Assessor Guidelines

Entity Selection Annotator GUI and pipeline revised to improve efficiency and

quality over previous years Enhanced annotators’ ability to select entities with rare slots

Slot Descriptions 15 slots revised after analysis of 2011 data

Slot Filling Annotation Guidelines for justification, duplicate fillers, normalization updated

for new pipeline and to address past challenges Assessment guidelines also revised to account for

justification Available at:

http://www.nist.gov/tac/2012/KBP/task_guidelines/index.htmlTAC KBP Evaluation Workshop – NIST, November 5-6, 2012

Existing SF Training Data (1)


Corpus Title (Dataset) Type LDC Catalog Language Size (Queries)5 GPE31 ORG17 PER

25 PER

25 ORG

50 PER

50 ORG

24 PER

8 ORG

30 PER

10 ORG

TAC 2010 KBP Evaluation Surprise Slot Filling Annotation

Evaluation LDC2010R22 English

10,416 Assessments

TAC 2010 KBP Training Surprise Task Slot Filling Annotation

Training LDC2010E52 English

TAC KBP 2009 Evaluation Slot Filling List Evaluation LDC2009E65 English

TAC 2009 KBP Assessment Results

Evaluation LDC2009E90 English

TAC 2010 KBP Training Slot Filling Annotation V2.1


TAC 2010 KBP Evaluation Slot Filling Annotation V1.1



Corpus Title (Dataset) Type LDC Catalog Language Size (Queries)

TAC 2010 Assessment Results V1.1

Evaluation LDC2010E61 English 25,511 Assessments

40 PER

10 ORG

50 PER

50 ORG

80 PER

20 ORG

TAC 2011 KBP English Regular Slot Filling Assessment Results V1.1

Evaluation LDC2011E88 English 28,041 Assessments

TAC 2011 KBP English Evaluation Temporal Slot Filling Annotation


TAC 2011 KBP English Training Temporal Slot Filling Annotation


TAC 2011 KBP English Evaluation Regular Slot Filling Annotation V1.2


Existing SF Training Data (2)

New SF Training & Eval Data


Corpus Title (Dataset) Type LDC Catalog Language Size (Queries)

25 PER

25 ORG

40 PER

40 ORG

40 PER

40 ORG

TAC 2012 KBP Spanish Slot Filling Training Queries and Annotations

Training LDC2012E68 Spanish/English

TAC 2012 KBP English Regular Slot Filling Assessment Results

Evaluation LDC2012E921 English22,885 Assessments

TAC 2012 KBP English Regular Slot Filling Evaluation Annotations

Evaluation LDC2012E91 English

TAC 2012 KBP Spanish Slot Filling Evaluation Annotations (UNRELEASED)

Evaluation LDC2012E920 Spanish/English

SF – Query Selection (1)


Stage 1: Select name strings and ref docs

Stage 2: Link namestrings to KB or

mark as NIL

Run named entity taggers over source corpora* Provides guided search through the corpus

Select entities and reference documents Non-confusable, productive namestrings

Identifiable and appropriate for Wikipedia Check KB node to ensure it wasn’t full

Rich entities (at least 2-3 fillers in the source corpus) Unique entities (fillers for under-utilized slots)

per:cause_of_death, per:charges, org:dissolved For Spanish queries, require mentions in both Spanish and

English source corpora

*Thank you to the track coordinators for providing tagger output


SF – Query Selection (2)

SF Annotator/Assessor GUIs


Annotation: For given entity, time-limited search

for fillers in corpus

Assessment: Check validity of asserted fillers & justification, create

equivalence classes

SF – Annotation/Assessment Approach

2012 Annotation For each query annotator spends up to 2 hours searching corpus

to identify fillers for targeted slot Quality control pass to flag and adjudicate fillers without adequate

source document justification and/or at variance with guidelines Added justification and duplicate, co-referenced fillers to process

2012 Assessment Assess validity of fillers and justification from humans and systems Create equivalence classes from fillers assessed as correct Quality control pass as with annotation


SF – Justification (1)

New requirement for 2012 SF annotation and assessment Intended to assist assessment by not requiring humans to review

whole documents to check validity of fillers Correct Justification

Includes all three pieces of information necessary to justify the entity/slot/filler relation

Does not include too much extraneous text


Barack Obama credits his family with his political success. Michelle Obama, now First Lady

President Obama and his wife, Michelle Obama

Barack Obama credits his family with his political success…

…[three intervening, unrelated paragraphs]……Michelle Obama, now First Lady

Wrong Justification No necessary information OR Slot Filler is wrong

Inexact Justification Filler is correct but justification is missing pieces of information or

includes too much extraneous text.

per/org:alternate_name Alternate name plus identifying information MTC Technologies <org:alternate_names> MTC


his wife, Michelle Obama

…[three unrelated paragraphs]…Barack Obama and his wife, Michelle Obama

…[three more unrelated paragraphs]…

MTC, based in Dayton, Ohio, is a supplier of logistical services to the Dept of Defense

SF – Justification (2)

Conclusions

2012 AchievementsNew language addedAdditional source data4 new corpora developedImproved annotation pipeline and GUIs allows for richer

and more unique queries with less annotator effort 2013 Goals

Further refine and improve pipeline and GUIsFurther develop guidelines for justificationFurther discussion of desired query qualities to fully

utilize new capabilities


Linguistic Resources for the 2012 TAC KBP Slot Filling Evaluations

Documents

Transcript of Linguistic Resources for the 2012 TAC KBP Slot Filling Evaluations