Linguistic Resources for the 2012 TAC KBP Slot Filling Evaluations
Joe Ellis (presenter), Brendan Callahan, Jonathan Wright, Stephanie Strassel
Linguistic Data ConsortiumUniversity of Pennsylvania, USA
Outline
English and Spanish source data Annotator and assessor guidelinesLabeled training and evaluation data
Annotation Tasks and MethodologiesEntity SelectionSlot Filling AnnotationSlot Filling Assessment
Linguistic Resources for 2012 Slot Filling
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Source Corpora – 2012
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Language Genre DocumentsBroadcast Conversation 17Broadcast News 665Conversation Telephone Speech 1
2,286,866
Web Text (2008 – 2009) 1,490,595
Spanish Newswire (2007 – 2010) 1,000,020
EnglishNewswire (2007 – 2010)
Language Source Nodes Catalog IDEnglish 2008 Wikipedia 818, 741 LDC2009E58
Source Corpora
SF Annotator/Assessor Guidelines
Entity Selection Annotator GUI and pipeline revised to improve efficiency and
quality over previous years Enhanced annotators’ ability to select entities with rare slots
Slot Descriptions 15 slots revised after analysis of 2011 data
Slot Filling Annotation Guidelines for justification, duplicate fillers, normalization updated
for new pipeline and to address past challenges Assessment guidelines also revised to account for
justification Available at:
http://www.nist.gov/tac/2012/KBP/task_guidelines/index.htmlTAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Existing SF Training Data (1)
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Corpus Title (Dataset) Type LDC Catalog Language Size (Queries)5 GPE31 ORG17 PER
25 PER
25 ORG
50 PER
50 ORG
24 PER
8 ORG
30 PER
10 ORG
TAC 2010 KBP Evaluation Surprise Slot Filling Annotation
Evaluation LDC2010R22 English
10,416 Assessments
TAC 2010 KBP Training Surprise Task Slot Filling Annotation
Training LDC2010E52 English
TAC KBP 2009 Evaluation Slot Filling List Evaluation LDC2009E65 English
TAC 2009 KBP Assessment Results
Evaluation LDC2009E90 English
TAC 2010 KBP Training Slot Filling Annotation V2.1
Training LDC2010E18 English
TAC 2010 KBP Evaluation Slot Filling Annotation V1.1
Evaluation LDC2010R11 English
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Corpus Title (Dataset) Type LDC Catalog Language Size (Queries)
TAC 2010 Assessment Results V1.1
Evaluation LDC2010E61 English 25,511 Assessments
40 PER
10 ORG
50 PER
50 ORG
80 PER
20 ORG
TAC 2011 KBP English Regular Slot Filling Assessment Results V1.1
Evaluation LDC2011E88 English 28,041 Assessments
TAC 2011 KBP English Evaluation Temporal Slot Filling Annotation
Evaluation LDC2011R40 English
TAC 2011 KBP English Training Temporal Slot Filling Annotation
Training LDC2011E49 English
TAC 2011 KBP English Evaluation Regular Slot Filling Annotation V1.2
Evaluation LDC2011R34 English
Existing SF Training Data (2)
New SF Training & Eval Data
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Corpus Title (Dataset) Type LDC Catalog Language Size (Queries)
25 PER
25 ORG
40 PER
40 ORG
40 PER
40 ORG
TAC 2012 KBP Spanish Slot Filling Training Queries and Annotations
Training LDC2012E68 Spanish/English
TAC 2012 KBP English Regular Slot Filling Assessment Results
Evaluation LDC2012E921 English22,885 Assessments
TAC 2012 KBP English Regular Slot Filling Evaluation Annotations
Evaluation LDC2012E91 English
TAC 2012 KBP Spanish Slot Filling Evaluation Annotations (UNRELEASED)
Evaluation LDC2012E920 Spanish/English
SF – Query Selection (1)
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Stage 1: Select name strings and ref docs
Stage 2: Link namestrings to KB or
mark as NIL
Run named entity taggers over source corpora* Provides guided search through the corpus
Select entities and reference documents Non-confusable, productive namestrings
Identifiable and appropriate for Wikipedia Check KB node to ensure it wasn’t full
Rich entities (at least 2-3 fillers in the source corpus) Unique entities (fillers for under-utilized slots)
per:cause_of_death, per:charges, org:dissolved For Spanish queries, require mentions in both Spanish and
English source corpora
*Thank you to the track coordinators for providing tagger output
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
SF – Query Selection (2)
SF Annotator/Assessor GUIs
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Annotation: For given entity, time-limited search
for fillers in corpus
Assessment: Check validity of asserted fillers & justification, create
equivalence classes
SF – Annotation/Assessment Approach
2012 Annotation For each query annotator spends up to 2 hours searching corpus
to identify fillers for targeted slot Quality control pass to flag and adjudicate fillers without adequate
source document justification and/or at variance with guidelines Added justification and duplicate, co-referenced fillers to process
2012 Assessment Assess validity of fillers and justification from humans and systems Create equivalence classes from fillers assessed as correct Quality control pass as with annotation
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
SF – Justification (1)
New requirement for 2012 SF annotation and assessment Intended to assist assessment by not requiring humans to review
whole documents to check validity of fillers Correct Justification
Includes all three pieces of information necessary to justify the entity/slot/filler relation
Does not include too much extraneous text
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Barack Obama credits his family with his political success. Michelle Obama, now First Lady
President Obama and his wife, Michelle Obama
Barack Obama credits his family with his political success…
…[three intervening, unrelated paragraphs]……Michelle Obama, now First Lady
Wrong Justification No necessary information OR Slot Filler is wrong
Inexact Justification Filler is correct but justification is missing pieces of information or
includes too much extraneous text.
per/org:alternate_name Alternate name plus identifying information MTC Technologies <org:alternate_names> MTC
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
his wife, Michelle Obama
…[three unrelated paragraphs]…Barack Obama and his wife, Michelle Obama
…[three more unrelated paragraphs]…
MTC, based in Dayton, Ohio, is a supplier of logistical services to the Dept of Defense
SF – Justification (2)
Conclusions
2012 AchievementsNew language addedAdditional source data4 new corpora developedImproved annotation pipeline and GUIs allows for richer
and more unique queries with less annotator effort 2013 Goals
Further refine and improve pipeline and GUIsFurther develop guidelines for justificationFurther discussion of desired query qualities to fully
utilize new capabilities
TAC KBP Evaluation Workshop – NIST, November 5-6, 2012
Top Related