Overview of TAC-KBP2015 Tri-lingual Entity Discovery and Linking

Heng Ji1, Joel Nothman2, Ben Hachey3, Radu Florian4

1 Computer Science Department, Rensselaer Polytechnic Institutejih@rpi.edu

2 Department of Computing and Information Systems, University of Melbournejoel.nothman@gmail.com

3 School of Information Technologies, University of Sydneyben.hachey@gmail.com

4 IBM TJ Waston Research Centerraduf@us.ibm.com

Abstract

In this paper we give an overview of theTri-lingual Entity Discovery and Linkingtask at the Knowledge Base Population(KBP) track at TAC2015. In this year weintroduced a new end-to-end Tri-lingualentity discovery and linking task whichrequires a system to take raw textsfrom three languages (English, Chineseand Spanish) as input, automaticallyextract entity mentions, link them to anEnglish knowledge base, and cluster NILmentions across languages. More entitytypes and mention types were also addedinto some languages. In this paper weprovide an overview of the task definition,annotation issues, successful methods andresearch challenges associated with thisnew task. This new task has attracteda lot of participants and has intriguedmany interesting research problemsand potential approaches. We believeit’s a promising task to be combinedwith Tri-lingual slot filling to form anew Tri-lingual cool-start KBP track inTAC2016.

1 Introduction

We have achieved some promising successes inEnglish Entity Discovery and Linking (EDL)in the previous years. However, for certainentities, a lot of new and detailed informationis only available in documents written in aforeign language for which there may be very fewlinguistic resources (annotated data, tools, etc.)

available. For example, when the Ebola outbreakstarted in 2014, news articles in Yoruba tend toreport the newest updates with many details suchas individual hospitals, researchers and local townnames. In contrast, news articles in English tendto only focus on general statistics such as thenumber of deaths, or non-local information suchas a foreign government’s reaction to the outbreak.Therefore, we believe it will be highly valuableto automatically link and fuse the knowledgeacross languages so we can construct a morecomplete profile in order to gain comprehensiveunderstanding of an entity or event. On theother hand, the new incident language (IL) oftenhas very low linguistic resources and data forrapid development of an EDL system. Butonce the cross-lingual links are built, we cantake advantage of the high-resources in English,including annotated data, gazetteers and richknowledge representations, and transfer them todevelop and enhance EDL in IL. Therefore,this year we extend EDL from mono-lingual tocross-lingual. An EDL system is required todiscover entity mentions from all three languagesinstead of one language, and link them to anEnglish Knowledge Base (KB) or cluster theminto NIL entities across languages.

Previous Entity Linking and EDL tasks mainlyfocused on three main types: Person, Organizationand Geo-political entities. Recent work (Xiaoand Weld, 2012; Lee et al., 2006) suggeststhat using a larger set of fine-grained types canlead to substantial improvement in downstreamNLP applications. We aim to gradually addnew entity types into the KBP program. Thisyear we add locations and facilities for all three

languages. This addition triggered some newresearch interests in fine-grained entity typing anduniversal entity schema discovery.

Finally, it’s valuable to cluster nominalmentions for creating new entries in KB. We alsoexplored person nominal mention extraction fromEnglish and identified some new challenges.

To summarize, compared to the KBP2014 EDLtask (Ji et al., 2014), the main changes andimprovement in KBP2015 include:

• Extended English EDL task frommono-lingual to tri-lingual;

• Added two new entity types - naturallocations (LOC) and facilities (FAC), for allthree languages;

• Added person nominal mentions for English;

• Prepared and used a new KB based onFreebase snapshot;

• Defined a new diagnostic task of EnglishEntity Discovery within the Cold-start KBPtrack.

The rest of this paper is structured as follows.Section 2 describes the definition of the fullTri-lingual EDL task and various diagnostic tasks.Section 3 briefly summarizes the participants.Section 4 highlights some annotation efforts.Section 5 summarize evaluation results and somegeneral progress report over years. Section 6summarizes new and effective methods, whileSection 7 provides some detailed analysis anddiscussion about remaining challenges. Section 8sketches our future directions.

2 Task Definition and Evaluation Metrics

This section will summarize the Tri-lingual EntityDiscovery and Linking tasks conducted at KBP2015. More details regarding data format andscoring software can be found in the task website1.

2.1 Full TaskGiven a document collection in three languages(English, Chinese and Spanish) as input,a tri-lingual EDL system is required toautomatically identify entity mentions from asource collection of textual documents in multiplelanguages (English, Chinese and Spanish),

1http://nlp.cs.rpi.edu/kbp/2015/

classify them into one of the following pre-definedfive types: Person (PER), Geo-political Entity(GPE), Organization (ORG), Location (LOC)and Facility (FAC), and link them to an existingEnglish Knowledge Base (KB), and clustermentions for those NIL entities that don’t havecorresponding KB entries. Figure 1 illustrates anexample for the full task.

Figure 1: Tri-lingual EDL Input/Output Example

Besides name mentions, person nominalmentions referring to specific, real-worldindividual entities should also be extracted fromEnglish. The system output includes the followingfields:

• system run ID;

• mention ID: unique for each entity mention;

• mention head string: the full head string ofthe entity mention;

• document ID: mention head start offsetmention head end offset: an ID for adocument in the source corpus from whichthe mention head was extracted, the startingoffset of the mention head, and the endingoffset of the mention head;

• reference KB link entity ID, or NIL clusterID: A unique NIL ID or an entity node ID,correspondent to entity linking annotationand NIL-coreference (clustering) annotationrespectively;

• entity type: GPE, ORG, PER, LOC, FACtype indicator for the entity;

• mention type: NAM (name), NOM (nominal)type indicator for the entity mention;

• confidence value.

Short name Name in scoring software Filter Key EvaluatesMention evaluationNER strong mention match NA span IdentificationNERC strong typed mention match NA span,type + classificationLinking evaluationNERLC strong typed all match NA span,type,kbid + linkingNELC strong typed link match is linked span,type,kbid Link recognition and classificationNENC strong typed nil match is nil span,type NIL recognition and classificationTagging evaluationKBIDs entity match is linked docid,kbid Document taggingClustering evaluationCEAFm mention ceaf NA span Identification and clusteringCEAFmC typed mention ceaf NA span,type + classificationCEAFmC+ typed mention ceaf plus NA span,type,kbid + linking

Table 1: Evaluation measures for entity discovery and linking, each reported as P , R, and F1. Span isshorthand for (document identifier, begin offset, end offset). Type is PER, ORG or GPE. Kbid is the KBidentifier or NIL.

2.2 Scoring MetricsTAC 2015 continues the 2014 measures forentity detection and linking (EDL) and itsdiagnostic (EL) variant, listed in Table 1. ELprovides gold standard mentions to systems,isolating linking and clustering performance. Thescorer is available at https://github.com/wikilinks/neleval.

2.2.1 Set-based metricsRecognizing and linking entity mentions can beseen as a tagging task. Here evaluation treats anannotation as a set of distinct tuples, and calculatesprecision and recall between gold (G) and system(S) annotations:

P =|G ∩ S||S|

R =|G ∩ S||G|

For all measures P and R are combined as theirbalanced harmonic mean, F1 =

2PRP+R .

By selecting only a subset of annotated fieldsto include in a tuple, and by including only thosetuples that match some criteria, this metric can bevaried to evaluate different aspects of systems (cf.Hachey et al. (2014) which also relates such metricvariants to the entity disambiguation literature).As shown in Table 1, NER and NERC metricsevaluate mention detection and classification,while NERL measures linking performance butdisregards entity type and NIL clustering. Inthe EL task where mentions are given, NERL isequivalent to the linking accuracy score reportedin previous KBP evaluations.

Results below also refer to other diagnosticmeasures, including NEL which reports linking(and mention detection) performance, discarding

NIL annotations; NEN reports the performanceof NIL annotations alone. KBIDs considersthe set of KB entities extracted per document,disregarding mention spans and discarding NILs.This measure, elsewhere called bag-of-titlesevaluation, does not penalize boundary errors inmention detection, while also being a meaningfultask metric for document indexing applications ofnamed entity disambiguation.

2.2.2 Clustering metricsAlternatively, entity linking is understood as across-document coreference task, in which theset of tuples is partitioned by the assigned entityID (for KB and NIL entities), and a coreferenceevaluation metric is applied. To evaluateclustering, we apply Mention CEAF (Luo, 2005),which finds the optimal alignment betweensystem and gold standard clusters, and thenevaluates precision and recall micro-averagedover mentions, as in a multiclass classificationevaluation. While other metrics reward systemsfor correctly identifying coreference withinclusters, a system which splits an entity intomultiple clusters will only be rewarded for thelargest and purest of those clusters. CEAFmperformance is bounded from above by NER,CEAFmC by NERC and so on.

Mention CEAF (CEAFm) is calculated asfollows. Let Gi ∈ G describe the goldpartitioning, and Si ∈ S the system, we calculatethe maximum score bijection m:

m = argmaxm

|G|∑i=1

∣∣Gi ∩ Sm(i)

∣∣s.t. m(i) = m(j) ⇐⇒ i = j

Then CEAFm is calculated by:

PCEAFm =

∑|G|i=1

∣∣Gi ∩ Sm(i)

∣∣∑|S|i=1 |Si|

RCEAFm =

∑|G|i=1

∣∣Gi ∩ Sm(i)

∣∣∑|G|i=1 |Gi|

As with set-based metrics, selecting a subset offields or filtering tuples introduces variants thatonly award score when, for example, the systemmatches the gold standard KB link or entitytype. Compared to KBP2014 CEAFm metrics, weadded two new enhanced variants:

• CEAFmC: adding type match into CEAFm

• CEAFmC+: combining CEAFmC and KBID matching, which can serve as anend-to-end metric for EDL to measure theoverall performance of extraction, linkingand clustering.

2.2.3 Confidence intervalsWe calculate c% confidence intervals for set-basedmetrics by bootstrap resampling documents fromthe corpus, calculating these pseudo-systems’scores, and determining their values at the100−c

2 th and 100+c2 th percentiles of 2500 bootstrap

resamples. This procedure assumes that andsystem annotates documents independently, andintervals are not reliable where systems use globalclustering information in their set-based output(i.e. beyond NIL cluster assignment). For similarreasons, we do not calculate confidence intervalsfor clustering metrics.

2.2.4 Weak boundary matchingSince the introduction of mention detection toTAC EDL in 2014, boundary errors in thedetection of mentions for linking are common.For some applications, it is appropriate to awardsystems for near matches on mention boundaries,which can be due to spurious ambiguity. In othercases systems’ performance could be improvedby fixing boundaries as a post-process, such asmerging adjacent mentions with the same entitylink, so we diagnostically consider these moreleniently.

To evaluate mention extraction performance, wealso added a variant for partial (weak) mentionboundary matching. It checks the number of

overlapped characters between a system generatedmention and a ground-truth mention. The impactof this strategy will be reported in section 5.6.

In addition, we have developed a method toalign unmatched system mentions to unmatchedgold mentions, for which there can be ambiguityin a coreference task such as NIL clustering.We outline the approach to finding a maximalalignment for evaluating under Mention CEAF,which will be described in full at a later venue:

1. Candidate mention pairs are grouped intopairs of gold and predicted entities.

2. The number of potential matches iscalculated for each gold-predicted entitypair. Since one gold mention may havemultiple unmatched mentions from thesame predicted entity, and vice-versa, this isnon-trivial.

3. These potential counts are added onto thecontingency matrix.

4. The maximum-scoring assignment ofpredicted to gold entities is found, as forCEAF calculation.

5. This can be used directly to calculateMention CEAF where potential alignmentsfor assigned entities are fixed.

6. Further boundary errors may be fixed byrepeating this process while disregardingentity pairs involved in the maximum-scoringassignment.

Note that duplicated system mentions for a singlegold mention (or vice-versa) will still result inprecision (or recall) errors, as at most one systemis fixed to each gold mention. Thus we alsoreport Mention CEAF under this optimal fixedspan condition (CEAFm-weak).

2.3 Diagnostic TasksIn order to investigate compare EDL’sperformance on various steps, languages, entitytypes, mention types, and check the progress overyears, we also allow systems to submit results ontheir favored combinations. For example, a teamcan choose to focus on 1-2 languages, or namementions, or linking task using perfect mentions.A perfect mention (’query’ as in Entity Linkingtasks in previous years) includes the followingfive fields:

• query id: A query ID, unique for each entitymention;

• mention: The full head string of the queryentity mention;

• docid: An ID for a document in the sourcecorpus from which the mention head wasextracted;

• The starting offset for the mention head;

• The ending offset for the mention head;

For example:〈query id=‘‘EDL15 ENG 0001"〉

〈name〉cairo〈/name〉〈docid〉bolt-eng-DF-200-192451-5799099〈/docid〉〈beg〉2450〈/beg〉〈end〉2454〈/end〉〈/query〉

The output format is the same as the full EDLtask.

3 Participants Overview

Table 2 summarizes the participants for theTrilingual EDL task. In total 10 teams submitted35 runs for the full task and 10 teams submitted 25runs for the diagnostic task (Entity Linking withperfect mentions as input).

Six teams performed linking across threelanguages in the full task, with other teams –including all those exclusively in the diagnostictask – linking one or two languages. Only fourteams identified person nominal mentions, thoughmost distinguished other entity types.

4 Data Annotation and Resources

The details of the data annotation for KBP2015are presented in a separate paper by the LinguisticData Consortium (Ellis et al., 2015). This yearwe used a new reference knowledge base derivedfrom BaseKB, a cleaned version of EnglishFreebase. The detailed statistics of the trainingand evaluation source collections are summarizedin Table 3 and Table 4 respectively.

Chinese Spanish English AllNews 84 82 85 251DF 63 47 83 193All 147 129 168 444

Table 3: Total # of Documents in Training Data

Chinese Spanish English AllNews 84 84 82 250DF 82 83 85 250All 166 167 167 500

Table 4: Total # of Documents in Evaluation Data

The corpus consists of topic-focused newsarticles and discussion forum posts published inrecent years, topically related comparable (butnon-parallel) across languages. LDC humanannotators selected documents so that a substantialamount of entities appear across two or threelanguages. Figure 2 and Figure 3 show that8.3% and 7.5% coreferential entities are acrosslanguages for training data and evaluation datarespectively, which provide great opportunitiesfor cross-lingual knowledge transfer (section 6.1).In the future systems may attempt working onstreaming news or social media data so theycan discover cross-lingual comparable documentsautomatically.

Figure 2: # of Coreferential Entities in TrainingData

IBM team reported some annotation errors onoverlapping mentions (Sil and Florian, 2015).There are also a few linking annotation errors. Forexample, in the following post: “”I am my ownman !” This is the phrase you will start hearingtoday from Jeb Bush as he tries to distance himselffrom his brother. But is he really his own man?”,the human annotator mistakenly linked the secondmention “man” to “George W. Bush”, while topsystems correctly linked it to “Jeb Bush”. LDCwill fix these errors and release updated annotation

Lan

guag

eE

ntity

and

men

tion

type

Gen

reTa

skTe

amA

ffilia

tion

CM

NE

NG

SPA

FAC

GPE

LO

CPE

RPE

R/N

OM

OR

GN

WD

F

ED

LIB

MIn

tern

atio

nalB

usin

ess

Mac

hine

sC

orpo

ratio

nX

XX

XX

XX

XX

XX

ED

LO

SUO

rego

nSt

ate

Uni

vers

ityX

XX

XX

XX

XX

XX

ED

LR

PIB

LE

ND

ER

Ren

ssel

aerP

olyt

echn

icIn

stitu

teX

XX

XX

XX

XX

XX

ED

Lhl

tcoe

Hum

anL

angu

age

Tech

nolo

gyC

ente

rofE

xcel

lenc

eX

XX

XX

XX

XX

XX

ED

LB

UPT

Team

Bei

jing

Uni

vers

ityof

Post

and

Tele

com

mun

icat

ion

XX

XX

XX

XX

XX

ED

LC

MU

Edv

isee

sC

arne

gie

Mel

lon

Uni

vers

ityX

XX

XX

XX

XX

XE

DL

HIT

SH

eide

lber

gIn

stitu

tefo

rThe

oret

ical

Stud

ies

XX

XX

XX

ED

LU

IC

CG

UIU

CC

ogni

tive

Com

puta

tion

Gro

upX

XX

XX

XX

XE

DL

ZJU

DC

DE

DL

Zhe

jiang

Uni

vers

ityX

XX

XX

XX

XE

DL

sdu

Shan

dong

Uni

vers

ityX

XX

X

EL

BU

PTTe

amB

eijin

gU

nive

rsity

ofPo

stan

dTe

leco

mm

unic

atio

nX

XX

XX

XX

XX

XX

EL

CM

UE

dvis

ees

Car

negi

eM

ello

nU

nive

rsity

XX

XX

XX

XX

XX

XE

LIB

MIn

tern

atio

nalB

usin

ess

Mac

hine

sC

orpo

ratio

nX

XX

XX

XX

XX

XX

EL

RPI

BL

EN

DE

RR

enss

elae

rPol

ytec

hnic

Inst

itute

XX

XX

XX

XX

XX

XE

LG

WU

Geo

rge

Was

hing

ton

Uni

vers

ityX

XX

XX

XX

XE

LH

ITS

Hei

delb

erg

Inst

itute

forT

heor

etic

alSt

udie

sX

XX

XX

XE

LSY

DN

EY

Sydn

eyU

nive

rsity

XX

XE

LZ

JUD

CD

ED

LZ

hejia

ngU

nive

rsity

XX

XX

EL

lvic

edl

Vis

ion

and

Con

tent

Eng

inee

ring

Lab

orat

ory

XX

XX

XX

XX

EL

sini

caA

cade

mia

Sini

caX

XX

XX

XX

XX

X

Tabl

e2:

Run

sSu

bmitt

edby

KB

P201

5Tr

iling

ualE

ntity

Dis

cove

ryan

dL

inki

ngPa

rtic

ipan

ts

Figure 3: # of Coreferential Entities in EvaluationData

packages.Finally, we also devoted a lot of time at

collecting related publications and tutorials 2,resources and softwares 3 to lower down the entrycost for EDL.

5 Evaluation Results

5.1 Overall PerformanceTable 5 and Table 6 summarize the resultsof the full EDL and the diagnostic EL tracksrespectively. We selected the best run fromeach system for comparison. For public releasepurpose we anonymized the team names. EnglishExtraction and Linking is much more difficultthan the same task in KBP2014, based on thecomparison of the same top systems acrosyears. Nevertheless, as a new cross-lingualKBP task, the overall results of both EDLand EL are very encouraging. Compared tolast year’s top mono-lingual English tracks (Jiet al., 2014), Tri-lingual EDL performance(CEAFm) is only 11% lower than mono-lingualEnglish EDL. The best Spanish-to-EnglishEntity Linking CEAFm score is improvedfrom 82.9% last year to 91.2% this year. Infact, cross-lingual setting has provided uniqueopportunities for cross-lingual inference. Somesystems have applied cross-lingual coreferenceand inference methods to transfer knowledgefrom one language to another and significantly

2http://nlp.cs.rpi.edu/kbp/2015/elreading.html3http://nlp.cs.rpi.edu/kbp/2015/tools.html

improved the performance of various components:entity typing, clustering and linking. Witheffective cross-lingual knowledge transfer RPIsystem (Hong et al., 2015) was able to achieve topCEAFmC score in Tri-lingual EL, even thoughit is based on an unsupervised linking algorithmwithout using any labeled data. More details willbe presented in section 6.1.

Comparing the full EDL performance(Figure 4) with the diagnostic EL performance(Figure 5), we can see the best CEAFmC scoredropped from 75.3% to 55.1% (system 1), andthe best CEAFmC+ score dropped from 72.4%to 59.4% (system 2). We can see that the bestTri-lingual mention extraction F-score is not great:72.4%; and the highest mention identificationrecall is 72.7%. This indicates that entity mentionextraction is a major challenge for EDL (detailedanalysis will be presented in section 7.1). Andthere is no single system that achieved the bestperformance at both mention extraction andlinking/clustering.

Figure 4: Tri-lingual EDL Performance

5.2 Progress from ACE to KBP MentionExtraction

Mention extraction (Florian et al., 2006; Li et al.,2014; Li and Ji, 2014; Lu and Roth, 2015) hasbeen an unsolved challenge since the AutomaticContent Extraction (ACE) program 4. But wehave been making great progress since then. IBMapplied their English ACE mention extractionsystem (Florian et al., 2006) to this year’sEDL evaluation data and only obtained 84.4%

4http://www.itl.nist.gov/iad/mig/tests/ace/

Syst

emNER

NERC

NERLC

KBIDs

CEAFm

CEAFm-weak

CEAFmC

PR

F1

PR

F1

PR

F1

PR

F1

PR

F1

PR

F1

PR

F1

TR

ILIN

GU

AL

379

.772

.776

.175

.969

.372

.467

.761

.864

.663

.265

.164

.164

.658

.961

.666

.560

.763

.562

.957

.460

.02

83.8

67.0

74.5

79.8

63.9

71.0

64.0

51.2

56.9

72.8

54.2

62.1

69.3

55.4

61.6

72.2

57.8

64.2

66.9

53.5

59.4

1172

.069

.370

.666

.063

.564

.753

.651

.552

.574

.354

.963

.155

.953

.854

.858

.556

.257

.353

.551

.552

.51

79.1

53.6

63.9

71.4

48.4

57.7

59.9

40.6

48.4

58.9

47.8

52.8

68.3

46.2

55.1

71.6

48.5

57.8

64.1

43.4

51.7

456

.263

.559

.750

.256

.753

.237

.942

.840

.227

.041

.532

.743

.549

.146

.147

.053

.149

.940

.545

.843

.012

72.7

55.3

62.8

64.2

48.9

55.6

44.2

33.6

38.2

45.5

48.3

46.9

54.7

41.6

47.3

56.9

43.4

49.2

49.5

37.7

42.8

CH

INE

SE

382

.277

.779

.979

.274

.876

.974

.470

.272

.268

.167

.567

.878

.273

.976

.080

.275

.877

.976

.171

.974

.011

78.8

68.7

73.4

74.7

65.1

69.6

61.2

53.3

57.0

83.7

47.7

60.8

71.0

61.8

66.1

75.2

65.5

70.0

68.2

59.4

63.5

285

.157

.868

.881

.955

.766

.365

.544

.553

.075

.139

.752

.076

.852

.262

.281

.755

.566

.174

.750

.860

.54

54.9

67.5

60.6

50.0

61.4

55.1

42.9

52.7

47.3

25.7

46.1

33.0

48.9

60.1

53.9

52.6

64.7

58.0

45.6

56.0

50.3

179

.438

.952

.270

.134

.346

.057

.928

.338

.057

.628

.538

.168

.033

.344

.773

.936

.248

.664

.831

.742

.612

76.2

36.9

49.7

72.0

34.9

47.0

45.8

22.2

29.9

65.2

28.4

39.6

52.8

25.6

34.4

55.4

26.8

36.2

50.6

24.5

33.0

1312

.310

.211

.22.

72.

32.

51.

00.

80.

966

.544

.053

.09.

17.

58.

224

.220

.222

.01.

51.

31.

4

EN

GL

ISH

383

.467

.574

.680

.064

.771

.569

.856

.562

.565

.365

.665

.476

.561

.968

.479

.264

.170

.874

.360

.166

.42

83.2

70.1

76.1

79.2

66.7

72.4

66.8

56.2

61.1

70.2

65.7

67.9

72.3

60.9

66.1

75.6

63.7

69.1

69.5

58.6

63.6

1174

.874

.074

.467

.166

.466

.753

.152

.552

.874

.858

.865

.865

.264

.564

.867

.867

.167

.561

.360

.660

.91

81.8

64.9

72.4

74.6

59.1

65.9

62.9

49.8

55.6

62.6

64.7

63.6

73.7

58.4

65.2

76.7

60.8

67.8

68.2

54.1

60.3

980

.560

.168

.871

.853

.661

.463

.447

.454

.263

.865

.164

.573

.955

.263

.276

.257

.065

.267

.150

.157

.412

77.1

69.1

72.9

67.4

60.5

63.7

50.1

45.0

47.4

48.2

62.9

54.6

63.6

57.1

60.2

66.1

59.3

62.5

56.9

51.0

53.8

582

.468

.975

.062

.752

.557

.153

.945

.149

.173

.564

.068

.474

.862

.668

.276

.864

.370

.057

.648

.252

.54

53.4

58.9

56.0

46.8

51.7

49.1

31.1

34.3

32.6

24.1

40.2

30.2

42.0

46.3

44.1

46.6

51.4

48.9

38.2

42.2

40.1

SPA

NIS

H

1480

.177

.278

.776

.173

.474

.770

.567

.969

.273

.074

.073

.575

.072

.373

.777

.274

.475

.872

.269

.670

.92

83.4

76.7

79.9

78.4

72.2

75.2

59.7

54.9

57.2

76.9

53.2

62.9

76.9

70.7

73.7

79.4

73.1

76.1

72.9

67.1

69.9

369

.477

.173

.163

.770

.767

.053

.959

.856

.754

.761

.157

.758

.464

.861

.461

.468

.264

.654

.960

.957

.74

67.5

68.1

67.8

60.2

60.8

60.5

45.6

46.0

45.8

39.6

37.9

38.8

54.1

54.5

54.3

56.8

57.3

57.0

49.7

50.1

49.9

171

.052

.160

.164

.247

.154

.354

.239

.745

.951

.241

.045

.563

.046

.253

.366

.048

.455

.958

.542

.949

.511

54.7

58.2

56.4

49.8

53.0

51.4

42.7

45.4

44.0

65.6

56.0

60.4

45.2

48.1

46.6

47.3

50.3

48.7

42.8

45.5

44.1

1258

.254

.356

.248

.845

.647

.127

.625

.826

.632

.946

.738

.647

.043

.945

.451

.748

.249

.940

.237

.538

.8

Tabl

e5:

Ove

rall

Ent

ityD

isco

very

and

Lin

king

Perf

orm

ance

Syst

emNERC

NERLC

NENC

KBIDs

CEAFm

CEAFmC

CEAFmC+

PR

F1

PR

F1

PR

F1

PR

F1

PR

F1

PR

F1

PR

F1

TR

ILIN

GU

AL

186

.988

.187

.571

.172

.271

.676

.071

.373

.665

.576

.470

.681

.182

.381

.774

.875

.975

.368

.269

.268

.72

85.8

85.9

85.9

74.1

74.2

74.2

69.5

78.4

73.7

77.1

78.9

78.0

82.9

83.0

82.9

75.0

75.1

75.1

72.3

72.4

72.4

389

.784

.086

.873

.568

.971

.177

.869

.673

.462

.971

.066

.773

.568

.971

.170

.065

.667

.765

.161

.063

.04

68.4

66.1

67.2

47.5

45.9

46.7

32.3

71.5

44.5

68.2

40.1

50.5

62.8

60.7

61.7

50.9

49.2

50.0

44.7

43.2

43.9

CH

INE

SE

394

.489

.491

.885

.480

.983

.189

.681

.485

.379

.776

.778

.292

.988

.090

.489

.384

.686

.983

.879

.481

.51

89.1

89.1

89.1

78.1

78.1

78.1

73.8

79.7

76.7

72.2

72.6

72.4

89.9

89.9

89.9

83.1

83.1

83.1

76.4

76.4

76.4

282

.882

.882

.871

.671

.671

.657

.978

.266

.580

.676

.278

.387

.587

.587

.574

.874

.874

.871

.371

.371

.34

70.6

70.6

70.6

54.9

54.9

54.9

33.6

82.4

47.7

77.5

43.7

55.9

76.8

76.8

76.8

60.5

60.5

60.5

53.2

53.2

53.2

631

.531

.531

.54.

84.

84.

88.

213

.810

.310

.112

.911

.366

.065

.966

.022

.322

.322

.34.

24.

24.

2

EN

GL

ISH

286

.887

.187

.073

.673

.873

.773

.675

.074

.371

.578

.474

.879

.980

.180

.074

.274

.474

.370

.570

.770

.63

88.6

81.3

84.8

69.2

63.5

66.2

76.9

58.0

66.1

57.3

72.8

64.2

80.1

73.5

76.6

75.7

69.4

72.4

67.7

62.1

64.8

185

.688

.186

.866

.168

.167

.176

.464

.870

.162

.779

.970

.379

.081

.480

.271

.373

.572

.463

.165

.064

.05

82.3

76.0

79.0

67.9

62.7

65.2

58.2

64.1

61.0

73.4

72.5

72.9

79.3

73.3

76.2

69.7

64.4

67.0

65.1

60.2

62.5

466

.862

.064

.340

.537

.538

.929

.161

.239

.461

.436

.245

.665

.961

.163

.449

.946

.348

.138

.235

.436

.77

61.6

56.9

59.1

46.2

42.7

44.4

32.9

66.6

44.1

60.2

28.5

38.7

60.0

55.5

57.7

46.3

42.8

44.5

39.9

36.9

38.3

861

.656

.959

.248

.644

.946

.742

.453

.647

.362

.657

.660

.052

.048

.049

.940

.337

.338

.734

.131

.632

.89

38.6

35.6

37.0

27.2

25.2

26.2

75.8

51.5

61.4

68.9

83.9

75.7

83.1

76.9

79.9

30.5

28.2

29.3

26.2

24.2

25.2

1038

.717

.424

.027

.112

.216

.963

.029

.339

.970

.933

.445

.480

.936

.450

.231

.714

.319

.725

.911

.716

.16

30.5

28.2

29.3

3.2

3.0

3.1

3.9

7.5

5.1

15.0

12.7

13.8

36.3

33.6

34.9

15.6

14.5

15.0

2.1

2.0

2.1

SPA

NIS

H

288

.988

.988

.980

.480

.480

.480

.587

.583

.884

.283

.583

.891

.291

.291

.283

.383

.383

.379

.679

.679

.61

86.1

86.1

86.1

71.5

71.5

71.5

78.5

76.2

77.3

63.9

74.9

69.0

80.6

80.6

80.6

74.5

74.5

74.5

66.6

66.6

66.6

383

.881

.082

.462

.260

.161

.266

.682

.773

.855

.960

.157

.972

.670

.171

.366

.163

.864

.957

.055

.056

.04

70.5

70.5

70.5

48.2

48.2

48.2

35.3

85.9

50.1

76.4

36.2

49.2

69.9

69.9

69.9

54.2

54.2

54.2

42.2

42.2

42.2

Tabl

e6:

Ove

rall

Ent

ityL

inki

ngPe

rfor

man

ce

Figure 5: Tri-lingual EL Performance

Precision, 49.9% Recall and 62.7% F-score,significantly lower than the F-score (71.5%) oftheir KBP2015 EDL system on the same data set.The low recall indicates the traditional mentionextractor might be “getting too old” for new namesdue to the fast language evolution and informalstyle of discussion forum posts.

Figure 6: English EDL Performance

5.3 Comparison on LanguagesFigures 7, 8, 9, 10 and 11 compare the breakdown scores for various languages. Overallwe don’t observe any language is particularlymore challenging than the other. There aretwo basic approaches to cross-lingual EDL: (1)Foreign Language EDL + Entity Translation;and (2) Full Document Machine Translation(MT) + English EDL. Most Chinese-to-Englishsystems adopted approach (1) while most

Spanish-to-English systems adopted (2),mainly because Spanish-to-English MT ismore mature than Chinese-to-English MT.UI CCG system (Sammons et al., 2015) reportedthat the F-score of English name tagging onSpanish-to-English MT output is 10% higherthan Spanish name tagging. In contrast,Chinese-to-English MT tends to miss andincorrectly translate many names. We will presentmore detailed error analysis in section 7.2.

Figure 7: Chinese EDL Performance

Figure 8: Spanish EDL Performance

5.4 Entity Types and Textual GenresThis year’s evaluation introduced new entity typesfor TAC KBP: FAC, GPE and LOC, as well asnominal mentions of PER entities. Figure 12 andFigure 13 show that overall NERL performanceclosely follows those for the frequent PER andGPE categories. Nominal mention detection is

Figure 9: English EL Performance

Figure 10: Chinese EL Performance

Figure 11: Spanish EL Performance

more challenging than name mention detection,and good nominal detection is not necessary for

strong end-to-end performance due to its relativelylow popularity in the source collection. RPISystem (Hong et al., 2015) designed specialheuristic rules and constraints to detect whether aperson nominal mention is generic or specific, andlink them to KB via within-document coreferenceresolution (details in section 6.2) and achievedthe best score on both nominal mention extraction(Figure 14) and linking (Figure 15). Of the namedentities, FAC performance is lowest by far.

Figure 12: EDL NERL F1 when selecting a subsetof annotations by text genre and entity type. NW =newswire, DF = discussion forum; FAC = facility,GPE = geopolitical entity, LOC = natural location,ORG = organization, PERNAM = person name,PERNOM = person nominal mention.

5.5 NIL and Non-NIL Comparison

Figures 16, 18, 19 and 17 compare the EDLperformance of NIL mentions and Non-NILmentions. Comparing NELC (Link recognitionand classification) and NENC (NIL recognitionand classification), we can see for English andChinese, NENC scores are significantly higher,which indicates that NIL mentions are not more

Figure 13: EDL NERL F1 when selecting a subsetof annotations by text genre and entity type. NW =newswire, DF = discussion forum; FAC = facility,GPE = geopolitical entity, LOC = natural location,ORG = organization, PERNAM = person name,PERNOM = person nominal mention.

Figure 14: Person Nominal Mention ExtractionPerformance

difficult to extract than Non-NIL mentions, andthe linking requirement brings extra challengesto Non-NIL mentions. In contrast, for Spanish,NIL mentions are more challenging than Non-NILmentions. Comparing NERL and NERLC scores,we can also see that mention typing accuracy isvery high for all three languages.

Figure 15: Person Nominal Mention LinkingPerformance

Figure 16: Tri-lingual EDL NIL and Non-NILPerformance Comparison

Figure 17: English EDL NIL and Non-NILPerformance Comparison

Figure 18: Chinese EDL NIL and Non-NILPerformance Comparison

Figure 19: Spanish EDL NIL and Non-NILPerformance Comparison

5.6 Impact of Boundary MatchWe evaluated the impact of the partial (weak)mention boundary matching strategy as describedin section 2.2.4. The score changes of Tri-lingualEDL are presented in Figure 20. We can see thatpartial matching can boost up to 13% F-score gainfor mention identification, and up to 9% F-scoregain for overall mention extraction (identificationand classification). The overall mention extractionand linking scores of two systems got a minordecrease (less than 1%) due to the propagation ofmention boundary detection errors to linking.

3 2 11 1 4 12Systems

0.02

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

F-Sco

re C

hange

NER

NERC

NERLC

Figure 20: Difference between Strong and WeakBoundary Match Scores

5.7 Linking: Are We Picking the SameLow-Hanging Fruits?

Figures 21, 22, 23 and 24 show the entitylinking performance on perfect mentions withoutconsidering typing. We can see that theperformance of top systems is very encouraging,generally around or above 80% for all threelanguages.

Figure 21: Tri-lingual Entity Linking on PerfectMentions

Two further questions naturally come upthough: (1) Are we solving the same easyproblems? and (2) Are we still facing the same

Figure 22: English Entity Linking on PerfectMentions

Figure 23: Chinese Entity Linking on PerfectMentions

Figure 24: Spanish Entity Linking on PerfectMentions

challenges? In order to answer these questions,we compare the overlapped instances from topthree entity linking systems: RPI, IBM and HITS.Figure 25 and Figure 26 depict the number ofoverlapped correctly linked mentions and errorsamong three systems respectively. We can drawthe following conclusions from these two figures:“3/4 happy families resemble each other, whileunhappy ones are 9/10 different.”.

Three systems faced the same challenges on thefollowing two cases:

• Rare entities. For example, in the followingsentence “The 3 High passes routes - I haveseen little information about Kongma La,Cho La and renjo La trails or the lodgeseither side of Cho La but lodges in the ThameValley have suffered - Lungden is closedand Thame itself has a lot of damage.”,all three systems mistakenly linked “Thame”to the town in Oxfordshire instead of the

Figure 25: Overlapped Correctly Linked Mentionsamong RPI, IBM and HITS

Figure 26: Overlapped Linking Errors among RPI,IBM and HITS

correct village in Nepal, because most ofother mentions in the context are NILs.

• Popularity Bias. Without knowing theglobal topic knowledge about a mention, allthree systems tend to mistakenly link it toa more popular entity. For example, in thefollowing sentence “Between the historicityof Clinton’s campaign, the Dems’ structural

advantage in the Electoral College, andClinton’s advantage with low-informationvoters, I think a lot of things out of his controlwould have to break right for Bush.”, allthree systems mistakenly linked “Bush” to“George W. Bush” instead of “Jeb Bush”.

• World Knowledge. In the followingsentence “The whole corruption scandalhas been ongoing for decades, its justconspicuous that ”suddenly” now the FBIand IRS decided to clamp down on theirregularities now that FIFA did not abide toWashington’s pressures.”, all three systemsmistakenly linked “Washington” to thecity instead of the correct entity “USgovernment”, due to the lack of knowledgethat a country’s capital is often used to referto its government.

Table 7 compares the overlapped instancesbetween RPI and IBM system outputs for threelanguages. We can see that two systems share themost similar strength and weakness for Chinesebecause of the unique challenges on morphresolution and name translation. In contrasttwo systems differ the most for English becauseRPI’s linker is unsupervised and IBM’s linkeris supervised. RPI’s entity linker tries to avoidusing labeled training data and relies only onco-occurred mentions for collective inference.This method has some advantage when indicativeentity mentions appear in local contexts. Forexample, In the following sentence “This ismore than the emergency needs of the UN’sWorld Fund Program (WFP) in the country,but we’re just talking about food here.”, RPIlinker successfully linked “WFP” to its correctKB entry “World Fund Program” which appearsright before the mention. In contrast, IBM linkermistakenly linked it to “World Food Programme”due to the noise introduced from other contextualwords such as “food”. On the other hand, IBMlinker benefited from its richer contextual featureextraction and supervised model for other cases.For example, in the following sentence “It’s beingreported that of the 21 people reportedly advisingJeb Bush, 19 are veterans of the first Bushadministration, the second Bush administration,or in a few cases, both.”, IBM linker successfullylinked “Bush” in “the first Bush administration”to “George H. Bush”, and “WFP” in “the secondBush administration” to “George W. Bush”, while

RPI linker mistakenly linked both of them to “JebBush” which was the only contextual mentionused for collective inference. Most systemsincluding HITS often make mistakes on mentionswith ambiguous entity types. For example, inthe following sentence “A statement from theWhite House also condemned the attack ”inthe strongest possible terms”.”, HITS systemmistakenly linked “White House” to the facilityentry in the KB. Due to the joint linking and typingmodel, both RPI and IBM systems successfullylinked it to the organization entity “ExecutiveOffice of the President of the United States”.

To summarize, top systems are facing differentchallenges, and have developed complementarytechniques. In the future we may considerassembling multiple linkers through effectivesystem combination and validation, similar to theslot filling validation task (Ji et al., 2010; Ji et al.,2011b).

Chinese Spanish English AllCorrect 82.0 77.1 72.2 76.2Error 18.2 9.9 8.5 11.3

Table 7: % of Overlapped Instances(Overlap/Union) between RPI and IBM ELsystem outputs

6 What’s New and What Works

6.1 Cross-lingual Knowledge Linking,Inference and Transfer

State-of-the-art EDL methods rely on entityprofiling and collective inference (Ji et al., 2014).In high-resource languages like English, we canuse some advanced knowledge representationssuch as Abstract Meaning Representation(AMR) (Banarescu et al., 2013) to effectivelyselect semantic neighbors for entity profilingand collaborators for collective inference (Panet al., 2015). However, such representationsare not available for low-resource languages.Moreover, generally other Natural LanguageProcessing (NLP) tools such as dependencyparsers in foreign languages also perform worsethan their counterparts in English. As a result,recent work on foreign language entity linkingattempted to shifted the focus on avoid excessivelinguistic analysis on the source documentsand fully leverage KB structure (Wang et al.,2015). Fortunately, this new Tri-lingual EDL

task provides new and unique opportunitiesfor cross-lingual knowledge transfer via entitytranslation, linking and inference. We couldbuild cross-lingual links via name translationin the source collection, and utilize the existingcross-lingual links between KBs, and then transferknowledge from a high-resource language to alow-resource language.

Figures 27 and 28 illustrate a motivatingexample. From the Chinese source collectionwe might not have enough resources to conductdeep understanding and thus can only extractco-occurrence based knowledge graphs, whichwould not be sufficient to correctly link twomentions “罗 姆 尼” and “保 罗” to theirreferent entities “Mitt Romney” and “Ron Paul”respectively in the English KB. But if wecan align these two mentions with two othermentions “Romney” and “Paul” in English sourcedocuments by name translation, then we canuse the rich knowledge representation fromEnglish documents (generated by AMR parserin this example) to infer the links to the KB,because “Romney” and “Paul” are connectedby a conjunction relation in the source sothey can be used as collaborators for collectiveinference; “Romney” and “Mitt Romney” sharemany neighbor nodes, and “Paul” and “Ron Paul”also share many neighbor nodes. Furthermore,the rich hyperlinks and cross-lingual links inmulti-lingual KBs can help jointly confirm thelinking decisions. RPI system (Hong et al.,2015) developed a joint mention extraction,translation and linking model, which enhanced thequality of mention boundary identification, typing,translation and linking simultaneously. HLT-COEsystem (Finin et al., 2015) also found thiscross-lingual coreference and inference approachcan greatly decrease the number of candidateclusters and thus reduce ambiguity.

Let’s look at another challenging examplebelow. From the Chinese discussion forum postitself, it would be almost impossible to resolvethe morphed mention “刀锋战士(Knife Warrior)”.However, if we can link the post to its topicallyrelated Chinese news article first, then we caneasily infer that “刀锋战士(Knife Warrior)”refers to “皮斯托瑞斯(Pistorius)” and should belinked to the South African sprint runner “OscarPistorius”.

• Chinese discussion forum post: 刀刀刀锋锋锋战战战士士士

Romney

Mitt Romney

coreference

Republican Party

nominate-01

MA

Massachusettscoreference

have-org-role-91 governor

Paul

Gettysburg College

visit-01

Texas

modify

GOPcoreference

conjunction: and

relocate-01

Merged KG From Source

(Romney)

· (Mitt Romney)

coreference

co-occurrence (Paul)

(Gettysburg College)

co-occurrenceco-occurrence

coreference

(Republican Party)

co-occurrence co-occurrence

(Massachusetts) (old Paul)

Figure 27: Multi-lingual Knowledge Graphs from the Source Collection

Ron_Paul

Mitt_Romney

hyperlink

Republican_Party

·langlink

·

hyperlink

party

partylanglink

(party)

(Presidential candidate) langlink

OfficeHolder

type

Multilingual KG From KBs

GOP

redirect

Massachusetts

orderInOfficelanglink

(Living Place)

Figure 28: Multi-lingual Knowledge Graphs from multi-lingual KBs

醒来发现女友不在身旁...刀刀刀锋锋锋战战战士士士 以过失杀人罪被判处5年监禁 (Knife Warriorfound his girl friend was not beside him whenhe woke up...Knife Warrior received a fiveyear prison sentence.)

• Chinese news document: 南非比勒陀利亚高等法院(The Pretoria High Court)21日宣布判处南非残疾运动员皮皮皮斯斯斯托托托瑞瑞瑞斯斯斯(OscarPistorius)5年有期徒刑。皮斯托瑞斯是南非著名的残疾人田径选手，有”刀刀刀锋锋锋战战战

士士士”之称。(On the 21th, the Pretoria HighCourt of South Africa sentenced the disabledsportsman Oscar Pistorius 5 year prison.Pistorius is a famous runner in South Africa,also named as ”Knife Warrior”.)

Figure 29 verifies this hypothesis. We cansee that the linking recall scores for entitiesacross three languages are much higher (up to38% difference) than entities that only appearin one language, because the linkers benefit

from the enriched contexts and KB propertiesacross languages and cross-lingual inferenceand propagation for linking decisions. Forcross-document NIL mention clustering, the recallscores of cross-lingual entities are lower thanmono-lingual entities due to the extra errors fromname translation.

Figure 29: Recall Comparison on Mono-lingualand Cross-lingual Entities

6.2 Tackling Nominal MentionsThis year, a EDL system is also required to extractperson nominal mentions referring to specific,real-world individual entities. There are two majorchallenges as follows.

(1) How to distinguish specific and genericnominal mentions. State-of-the-art three-classrealis classification accuracy for events is stillbelow 50% (Hong et al., 2015). Similarly, weoften need to analyze the entire context sentenceto determine whether a nominal entity mention isgeneric or specific. RPI system (Hong et al., 2015)encoded heuristic rules to detect the followingcategories:

(a) Hypothetical: can be detected usingkeywords such as indefinite articles (e.g., “a/an”)and conditional conjunctions (e.g., “if”). Forexample, the nominal mentions in the followingsentences are all generic:

• Apparently, Bibi assumes that the next USPresident will see things much more hisway than does Barack Obama. If H is thenominee, he may be right.

• because it is so widely presumed that H is theinevitable candidate.

• And if something happened to Hillary hecould become president.

(b) Subjective Mood: can be detected bydiscourse structures. For example, “VicePresident” in the following sentence is generic:

• while President Hillary Clinton lived in theWhite House ... Vice President Bill C wouldreside at N.

(c) Generic Referent: many cases can bedetected by keywords such as “should” and “a”:

• instead it’s the US President who shouldtake the abuse and apologize for having abackbone.

However, some cases need background knowledgeto infer a nominal mention is generic. Forexample, only if we know the following sentenceis quoted from a legal document, we can infer that“President” is a generic mention:

• ”which lists they shall sign and certify,and transmit sealed to the seat of thegovernment of the United States, directed tothe President of the Senate.”

This approach achieved about 46% F-score forperson nominal mention extraction from Englishtraining data.

(2) How to link or cluster these nominalmentions. In order to link a nominal mentionto KB, or assign it to a NIL cluster), the mosteffective approach is to apply within-documentcoreference resolution to resolve it to aname mention. Although state-of-the-artwithin-document coreference resolutionperformance for nominal mentions is stillquite low, linking each identified person nominalmention to its closest person name mention canyield 67% accuracy (Hong et al., 2015).

6.3 Global Knowledge Derived from KB

Due to the lack of resources in foreign languages,Tri-lingual EDL systems have been trying to bemore creative at utilizing global knowledge inthe English KB. For example, RPI system (Honget al., 2015) leveraged knowledge graph basedembeddings, developed an entropy based measureto quantify the discriminative power of eachlink type in the KB. IBM system (Sil andFlorian, 2015) proposed a global feature basedon computing pointwise mutual information forthe Wikipedia categories of consecutive pairsof entities, and significantly enhanced linkingaccuracy.

In addition, many systems includingHITS (Heinzerling et al., 2015), LVIC (Besancon

et al., 2015), IBM (Sil and Florian, 2015) andRPI (Hong et al., 2015) utilized entity linkingresults as feedback to map the entity properties inKB to one of the five types defined in KBP. RPIsystem utilized Abstract Meaning Representation(AMR) corpus (Banarescu et al., 2013) whichcontains over 100 fine-grained entity types andhuman annotated KB titles. DBPedia5 alsoprovides rich types for each page. Therefore,they generated a mapping table between AMRtype and DBPedia rdf:type (e.g., university -TechnicalUniversitiesAndColleges).The typing F1 score is 93.2% for perfectmentions in English training data. HITSsystem (Heinzerling et al., 2015) removed allsense annotations whose entity types from nametaggers don’t agree with the rdf:type in KB.

6.4 Name Translation Mining

In the pilot Tri-lingual EDL study conducted inthe DEFT program earlier this year, we foundthat a pipeline of foreign language ED + NameTranslation + Linking is more effective thanfull document Machine Translation + EnglishEDL. Therefore, Name Translation becomes acritical component. Despite that many challengesstill remain for name translation (section 6),teams have been developing creative approachesto automatically mine name translation pairs.For example, RPI system has developed anovel unsupervised approach to construct andalign bursty knowledge graphs from multi-lingualstreaming data, incorporating novel criteria basedon multi-dimensional clues from pronunciation,translation, burst, neighbor and graph topologicalstructure. This approach was able to mine manyhigh-quality name translation pairs.

Many Chinese news documents include nametranslation pairs in the parenthesis pattern. Forexample, we can mine from many pairs from thenews document below: “鲍里斯·Y·涅姆佐夫(Boris Y. Nemtsov)”, “俄罗斯内政部(InteriorMinistry)”, “鲍 里 斯·N·叶 利 钦(Boris N.Yeltsin)”, “弗拉基米尔·V·普京(Vladimir V.Putin)”, “国际文传电讯社(Interfax)”, “对话(Sobesednik)”, “福布斯(Forbes)”, “保罗·克列布尼科夫(Paul Khlebnikov)”, “安娜·波利特科夫斯卡娅(Anna Politkovskaya)” and “纳塔利娅·埃斯蒂米洛娃(Natalya Estemirova)” usingthe methods described in previous work (Lin et al.,

5http://dbpedia.org

2008; Ji et al., 2009).

• 周五深夜，俄罗斯知名反对派领袖鲍鲍鲍里里里斯斯斯···Y···涅涅涅姆姆姆佐佐佐夫夫夫(Boris Y. Nemtsov)在莫斯科市中心遭枪击身亡，遇难地距离克里姆林宫的围墙几步之遥。 周六凌晨1点刚过不久，俄俄俄罗罗罗斯斯斯内内内政政政部部部(Interior Ministry)证实了涅姆佐夫遇刺一事。享年55岁的涅姆佐夫曾是鲍鲍鲍里里里斯斯斯···N···叶叶叶利利利钦钦钦(Boris N. Yeltsin)的第一副总理，后参与组织反对弗弗弗拉拉拉基基基米米米尔尔尔···V···普普普京京京(Vladimir V. Putin)的示威活动。 国国国际际际文文文传传传电电电讯讯讯社社社(Interfax)援引一名警方知情人士称，这似乎是一次买凶杀人。 本月早些时候，涅姆佐夫曾告诉《《《对对对话话话》》》(Sobesednik)杂志，他想过俄罗斯总统弗弗弗拉拉拉基基基米米米尔尔尔···V···普普普京京京(Vladimir V. Putin )可能会杀掉自己，不过，他似乎并没有把这个想法当回事。他说他的母亲比他忧虑的多。2004年，《《《福福福布布布斯斯斯》》》(Forbes)杂志的保保保罗罗罗···克克克列列列布布布尼尼尼科科科夫夫夫(Paul Khlebnikov)遭枪击身亡；2006年，以反对车臣 战争的激烈论战而闻名的安安安娜娜娜···波波波利利利特特特科科科夫夫夫斯斯斯卡卡卡娅娅娅(Anna Politkovskaya)遭到枪杀。2009年，纳纳纳塔塔塔利利利娅娅娅···埃埃埃斯斯斯蒂蒂蒂米米米洛洛洛娃娃娃(Natalya Estemirova)在北高加索遭遇绑架，并被枪击身亡。

6.5 Fine-grained Entity TypingAdding new entity types into KBP2015 hastriggered some new research efforts onfine-grained entity typing. RPI system tookone step further to discover new entity typesautomatically. They start from learning generalembeddings for each entity mention, compose theembeddings of specific contexts using linguisticstructures, link the mention to knowledge basesand learn its related knowledge representations.Then they developed a novel joint hierarchicalclustering and linking algorithm to type allmentions using these representations. Thetypes of all entity mentions are automaticallydiscovered based on a set of clusters, which cancapture fine-grained types customized for anyinput corpus. This framework doesn’t rely onany annotated data, predefined typing schema,or hand-crafted features, therefore it can bequickly adapted to a new domain, genre andlanguage. For example, RPI Tri-lingual EDLsystem can be easily adapted to the biomedicaldomain, by replacing Freebase with biomedical

ontologies. For example, given a sentence“The effects of the MEK inhibitor on totalHER2 , HER3 and on phosphorylated pHER3were dose dependent.”, it can link “HER2”to “ERBB2” in BioPortal and extract the type‘Proto-Oncogenes→Oncogenes→Genes→GenomeComponents→Genome→Phenomena andProcesses’ as the type for this entity mention.

6.6 Portability for a New LanguageAdding foreign languages into the EDL taskhas shifted some research focus from quality toportability. Supervised learning usually producesbetter Entity Linking results than unsupervisedlearning (Ji et al., 2011a). However, theysuffered from the high cost of large-scale manualannotation - 90% linking accuracy requires about20,000 query mentions for training. This yearmore systems have been seeking new methodsfor rapid low-cost development or adaption ofEDL techniques to a new language. Forexample, RPI’s unsupervised entity typing andlinking framework (Hong et al., 2015) was ableto apply a new surprise language overnight.The mention-level typing accuracy with perfectboundary is very promising: 85.42% for Hausaand 72.26% for Yoruba. IBM system (Sil andFlorian, 2015) trained their linking componentfrom English Wikipedia and thus it can be adaptedto new languages without re-training.

7 Remaining Challenges

7.1 Mention ExtractionRegardless of the progress since ACE, theperformance of mention extraction is still notsatisfactory. The best KBP2015 Tri-lingualmention extraction system’s F-score is 72.4%, andthe best English mention extraction F-score is76.1%. In the following we will highlight someremaining major challenges across top systems.

7.1.1 OOV Mention IdentificationThe highest mention identification recall is only72.7%. Some English error types include:

• Informal Nominals: Most missing errors arecaused by the mentions that rarely appearedin traditional newswire training data. Forexample, in a discussion forum post “OK,what in cheney’s background makes you thinkhe’s the puppeteer and bush is the puppet?”,the person nominal mention “puppeteer”

doesn’t appear in English training data soit’s very difficult for a mention detectorto identify it unless a good coreferenceresolver can be applied to link it to “he”.Likewise “mastermind” in the followingsentence is a person nominal mention and canbe identified by resolving it to “MohamedMohamud”: “Kenyan enyan authorities alsoput a $220,000 (200,000 euro) bounty onthe head of the alleged mastermind of theattack, Mohamed Mohamud, - also known asDulyadin Gamadhere - who is believed to bein Somilia.”. Probably it’s worth re-visitingsome previous work about joint inferencebetween coreference and name tagging (Ji etal., 2005).

• Abbreviations: Abbreviations, especiallysingle-letter ones, are difficult to identify. Forexample, in a discussion forum post “Eventhe Bush family knew W needed help movinghis lips.”, “W” should be identified as aperson name that refers to George W. Bush.

• Names embedded in URLs: It’s alsodifficult to identify uncommon namesfrom URLs, such as “netanyahus” in“http://news.yahoo.com/liberal-israelis-netanyahus-win-reality-check-115401998.html”. Perhaps next year weshould remove this requirement

Typical Chinese error types include:

• Code-switch: Chinese documents ofteninclude English names, which need tobe identified by specific patterns. Forexample, many systems failed to identifythe organization name “Dahabshiil” from thefollowing sentence: “Dahabshiil就是名单上一家汇款公司，它的业务涉及整个非洲之角地区。”.

• Ancient entities: Chinese discussion forumposts often quote ancient person namesmentioned in a document written in classicalChinese. For example, the followingdiscussion forum post tries to explain“revolution” by quoting a sentence from abook I ching written during 1000750 BC: “详细解释：出处及演变“革命”一词的古义是变革天命，最早见于《周易·革卦·彖传》：“天地革而四时成，汤汤汤

武武武革命，顺乎天而应乎人。” (Detailedexplanation: the origin and evolution of theword “revolution” are from its ancientmeaning, change destiny, which appearedearliest in ”I ching. Gegua. Tuanzhuan”:”The timing is right for the revolution ofthe Heaven and the Earth, Tang againstWu’s revolutions obey the will of Heavenand be in harmony with men.”)”, from thisquoted sentence a name tagger needs toidentify “汤 (Tang)” and “武 (Wu)” as personnames, referring to two historical figures “商汤 (Shang Tang)” and “周武王(Zhou WuKing)” respectively.

• Morphs: In the KBP2014 EDL overviewpaper (Ji et al., 2014) we pointed out aunique challenge from Chinese discussionforum - Entity ”Morphs”, a special type offake alternative names, to achieve certaincommunication goals such as expressingstrong sentiment or evading censors. Thisyear we extended the Chinese track fromEntity Linking to Entity Discovery andLinking. Compared to linking, it’s even moredifficult to identify these morph mentionsfrom texts (Zhang et al., 2015). In KBP2015EDL training data, about 16% mentionsare morphs. For example, in a discussionforum post “所以，我觉得，乡港人还是太没有气魄了，只占领*中*环算什么？ (So I think Hong Kong peopleare still not brave, what’s the big dealto occupy the Central?)”, “乡港 (XiangGang)” is a morph referring to “香港 (HongKong)”, and “*中*环 (Central)” is anothermorph referring to “*中*环 (Central)”. Itrequires an EDL system to incorporate newtechniques such as new name discovery or amorph identified trained specifically from alarge amount of Chinese social media data.

7.1.2 Boundary Errors due to InformalContexts

Compared to newswire, the top EDL systemsmade a lot more mention boundary errors ondiscussion forum due to its informal nature. Forexample, one of the top systems mistakenlyidentified “Pres Obama” as a person name from“while Pres Obama and Holder are willingto give them a pass.” Some effective textnormalization techniques might be worth adding

before mention extraction, e.g., to expand “Pres”to “President”. In another post: “Let’s giveObama a THIRD TERM as President Have Pelosiand Reid introduce a Bill to bomb some placethat republicans hate.”, many systems mistakenlyidentified the informally capitalized word “Have”as part of an incorrect name mention “HavePelosi”.

7.1.3 Joint Linking and Typing: When toTrust Whom More?

Despite the great success achieved by jointmodeling of entity typing and linking, there isstill no clear and elegant solution to automaticallydecide when to trust typing and when to trustlinking. Most EDL systems chose to uselinking feedback to override typing results fromname tagging, with the assumption that linkingaccuracy is higher than typing. However, thisstrategy may introduce errors for some highlyambiguous mentions, especially in discussionforum posts. For example, in the followingsentence “His stands on fracking, the TPP,indefinite detention, the Patriot Act, torture,war crimes, drone killings, persecution ofwhistle-blowers and journalists, etc., precludehim from being a ”Populist President”.”, “TPP”refers to “Trans-Pacific Partnership”. However,most top EDL systems mistakenly linked itto “Comandante FAP Guillermo del CastilloParedes Airport (IATA: TPP)” and thus labeledit as a facility. Therefore the current jointinference approaches still need to be improved byintegrating more reliable confidence estimation.

7.2 Machine Translation and EntityTranslation

Most Spanish-to-English EDL systems usedGoogle Translation service or Bing Translationservice to translate Spanish documents intoEnglish and then apply English EDL to theMT output. Compared to Spanish-to-EnglishMachine Translation (MT), state-of-the-artChinese-to-English MT performance is still notsatisfactory. For example, the following showsthe translation results from various MT systems:

• Chinese: 经过一个展厅，从条幅上看，写的是敬贺本本本焕焕焕老和尚百年寿辰书画展，这才知道老和尚生于1907年，已经年满百岁。

• Reference Translation: After passing

through an exhibition hall, from the banner,we could see that it was a painting andcalligraphy exhibition to celebrate the oldmonk Ben Huan’s centennial birthday. Notuntil then we knew that this old monk wasborn in 1907 and so he is already 100 yearsold.

• Research MT System 1: After a Gallery,from banner, write the Worship Celebration,the old monk 100 years birthday dynastiesthat old monk who was born in 1907, has over100 years old.

• Research MT System 2: after an exhibitionhall, from the point of view of scrolls, writingis hwan old monk centennial birthday ofpainting and calligraphy, he did not know ituntil this old monk was born in 1907 years, isalready 100 years old.

• Google MT: After an exhibition hall, frombanners point of view, this is written, He JingHuan birthday centuries old monk paintingand calligraphy exhibition, to realize that theold monk born in 1907 , has over a hundred .

Both research MT systems are state-of-the-artstatistical models trained from 2 million sentencepairs. We can see that even though the generaltopics of this sentence are correctly translated,none of these three MT systems translated theperson name “本焕(Ben Huan)” correctly. Ourprevious analysis (Ji et al., 2009) showed thata state-of-the-art Chinese-to-English MT systemmis-translated about 40% person names.

Therefore, most Chinese-to-English EDLsystems have chosen an alternative pipeline:Chinese EDL + Entity Translation. Unfortunately,Chinese-to-English Entity Translation is nota trivial task either. Our KBP2011 overviewpaper (Ji et al., 2011a) presented the detailederror distributions for Entity Translation. Manyof these challenges still remain, especiallywhen the Chinese name is composed based onmeanings while the English translation is basedon pronunciation. For example, “上海国际化自贸区” should be translated and linked to“Causeway Bay” in the English KB, but its literaltranslation “Shanghai International Free TradeZone” doesn’t share any words with the officialtranslation. We need to improve name structureparsing, and develop some automatic mechanism

to decide when to translate or transliterate aname or a name component. Some systemssuch as RPI (Hong et al., 2015) have developedbetter name transliteration models. But thesemodels usually rely on a large amount of namepairs, which might not be adaptable for otherlow-resource foreign languages. It would be morepromising to focus on automatic name translationmining techniques.

7.3 Entity Linking7.3.1 Abbreviations and NicknamesIn Chinese, GPE names are often abbreviated assingle characters, such as “美” for “美国(UnitedStates)” and “中” for “中国(China)”. Thesesingle characters are highly ambiguous in variouscontexts. For example, “美” can also be a commonadjective which means “beautiful”.

For many abbreviations and nicknames, an EDLsystem usually should assign more weight tocontext similarity than popularity. For example,in the following post “香港人都支持占中?(Doall of Hong Kong people support Occupy theCentral?)”, the GPE mention “中” should belinked to “Central”, the central business districtof Hong Kong, instead of the more popular entry“China”. And in “First, as much as Bibi saysIsrael will go it alone if it has to, I doubt thatit would/could.”, we should link “Bibi” shouldbe linked to the Israeli politician “BenjaminNetanyahu”, instead of the more popular Swedishactress “Bibi Andersson”.

7.3.2 Re-visit “Collaborators” in CollectiveInference

State-of-the-art Entity Linking techniques relyon the idea of “Collective Inference”, wherea set of mention collaborators are linkedsimultaneously by choosing an “optimal” ormaximally “coherent” set of named entity targets -one target entity for each mention in the coherentset. Many existing methods extract“collaborators”based on co-occurrence (Ratinov et al., 2011;Wang et al., 2015), topic relevance (Cassidy etal., 2012), social distance (Cassidy et al., 2012;Huang et al., 2014), dependency relation (Linget al., 2014), or a combination of these throughmeta-paths (Huang et al., 2014). However,two mentions are qualified as collaborators forcollective inference not because they are involvedin a syntactic structure. Rather it’s becausethey are often involved in some specific types of

relations and events. Some other work tried torestrict collaborators that bear a specific type ofrelation (Cheng and Roth, 2013). But high-qualityrelation/event extraction (e.g., ACE) is limited toa fixed set of pre-defined types. One potentialsolution is to construct a background knowledgebase in a never-ending way to gather relationsand events for each entity in real time, thenwe can infer collaborators from this backgroundknowledge base.

7.3.3 Knowledge RepresentationRecent advances in rich knowledgerepresentations such as AMR have greatlypromoted unsupervised entity linking (Pan etal., 2015). However, in some cases even AMRcannot capture implicit relations among entitiesand concepts. For example, in the followingsentence “The Stockholm Institute stated that23 of 25 major armed conflicts in the worldin 2000 occurred in impoverished nations.”,the concept “armed” is crucial to determinethat “The Stockholm Institute” should be linkedto “Stockholm International Peace ResearchInstitute” instead of “Stockholm Institute ofEducation”. However, the AMR graph forthis sentence (as depicted in Figure 30) cannotcapture the semantic connections between “TheStockholm Institute” and “armed”. Entity Linkingis likely to benefit from adding even richer typesof nodes and edges, and Cross-sentence nominaland pronoun coreference resolution into AMR.

Figure 30: AMR graph for the example sentence.

7.3.4 Background KnowledgeMost discussion forum posters assume the readersalready know the on-topic entities and events, andthus they don’t bother to elaborate the background

for these target entities. Also they tend to use shortand informal mentions for efficient discussions.As a result, directly comparing the contexts of amention in a post with a candidate KB entity’s textdescription is often not sufficient. For example,in order to figure out that “Gulf ” in the followingpost “I went to youtube and checked out theGulf oil crisis: all of the posts are one monthold, or older…” refers to “Gulf of Mexico”, weneed to know it suffered a catastrophic explosion.For the following post “What words about Bush,Cruz, Romney, Carson, Walker?”, we first needto know it’s about presidential election, then link“Cruz” to “Ted Cruz” instead of the more popularSpanish actress “Penélope Cruz”. Similarly, weneed to know the following post “Whitewater,Monica, B, infidelity, Foster, health care reform,sniper fire, Sir Edmund Hillary, head trauma,I war vote, support for the Iraq war, email,servers, lesbians........” is talking about Clintonin order to link “Whitewater” to “WhitewaterDevelopment Corporation”, and “Monica” to“Monica Lewinsky”. An entity linker needs toautomatically construct a background knowledgebase as a bridge between the source collection andthe KB.

7.3.5 Commonsense KnowledgeEntity linkers are still lack of exploitingcommonsense knowledge. For example, topsystems mistakenly link “Kenyatta” in thefollowing sentence “In his first televised addresssince the attack ended on Thursday, Kenyattacondemned the ”barbaric slaughter” and askedhelp from the Muslim community in rootingout radical elements. ” to “Jomo Kenyatta(1891-1978)”, because it share the same lastname as the correct entry “Uhuru Kenyatta”, andboth served as the President of Kenya. But fromthe post we can clearly see the target entity isstill alive because he made an announcement.Other types of comprehensive commonsenseknowledge is required to disambiguate somedifficult cases. For example, we need to know thata capital city can be used to refer to a country’sgovernment, in order to link “Washington” to“Federal government of the United States” inthe following post “Millions of Americans wentto war for America, and came back broken orotherwise gave up a lot, and now we look to takea huge chunk of their hide because Washington nolonger works.”.

7.3.6 Morph DecodingMorphs remain the most challenging mentions inChinese Entity Linking. There are many differenttechniques that have been used to encode thesemorphs (Zhang et al., 2014), therefore surfacefeatures and context similarity are far from enoughto decode them successfully. For example, “天朝(Heaven Dynasty)” is a morph created for“China” due to its long history. “帖木儿(PostWood Er)” is created to refer to “Genghis Khan”because its pronunciation “Tie Mu Er” is closeto his born name “Temüjin”. In English, 2%mentions are morphs. For example, in “Theypassed a bill, and Christie the Hutt decides he’sstull sucking up to be RomBot’s running mate.I think the Good Doctor is too crazy to hang itup.”, “Christie the Hutt”, “RomBot” and “GoodDoctor” refer to “Chris Christie”, “Mitt Romney”and “Ron Paul” respectively.

8 Looking Ahead

The new Tri-lingual EDL task has created manyinteresting research problems and new directions.In KBP2016 we will consider the followingpossible extensions and improvement:

• Combine with tri-lingual slot filling andtri-lingual event extraction to form up anend-to-end cool-start tri-lingual KBP task;

• Target at a larger scale data processing, byincreasing the size of source collections from500 documents to 10,000 documents;

• Add EDL for individual specific nominals forPER, GPE, ORG, LOC and FAC entities intoall three languages; This addition is likelyto promote two research directions: (1) Itintroduces a new definition of mentions fromthe end usage of KB construction; (2) Itmay promote within-document coreferenceresolution research which is currently abottleneck for all KBP tracks (EDL, SlotFilling, Event KBP);

• Add more fine-grained entity types, or allowEDL systems to automatically discover newentity types; We may start by adding Weapon,Vehicle, Commodity and other Productsubtypes as defined in AMR (Banarescu etal., 2013) such as work-of-art, picture, music,show, broadcast-program, publication, book,newspaper, magazine and journal;

• Add a new task of EDL assembling;

• Add streaming data into the sourcecollection;

• Perhaps replace Spanish with a newlow-resource language for whichfull-document MT techniques are lessmature.

Acknowledgments

Special thanks to Hoa Trang Dang (NIST)for coordinating and overseeing this track.Thanks Joe Ellis (LDC) for providing datastatistics. Thanks Xiaoman Pan (RPI) for hisassistance on result analysis. Heng Ji’s workhas been supported by U.S. DARPA DEFTProgram No. FA8750-13-2-0041, ARL NS-CTANo. W911NF-09-2-0053, and NSF IIS-1523198.The views and conclusions contained in thisdocument are those of the authors and shouldnot be interpreted as representing the officialpolicies, either expressed or implied, of theU.S. Government. The U.S. Government isauthorized to reproduce and distribute reprintsfor Government purposes notwithstanding anycopyright notation here on.

ReferencesL. Banarescu, C. Bonial, S. Cai, M. Georgescu,

K. Griffitt, U. Hermjakob, K. Knight, P. Koehn,M. Palmer, and N. Schneider. 2013. Abstractmeaning representation for sembanking. In Proc.Linguistic Annotation Workshop.

R. Besancon, H. Daher, H. L. Borgne, O. Ferret,A. Daquo, and A. Popescu. 2015. Cea listparticipation at tac edl english diagnostic task. InProc. Text Analytics Conference (TAC2015).

T. Cassidy, H. Ji, L. Ratinov, A. Zubiaga, andH. Huang. 2012. Analysis and enhancement ofwikification for microblogs with context expansion.In Proceedings of COLING 2012.

X. Cheng and D. Roth. 2013. Relationalinference for wikification. In Proceedings of the2013 Conference on Empirical Methods in NaturalLanguage Processing.

J. Ellis, J. Getman, D. Fore, N. Kuster, Z. Song,A. Bies, and S. Strassel. 2015. Overview oflinguistic resources for the tac kbp 2015 evaluations:Methodologies and results. In Proc. Text AnalysisConference (TAC2015).

T. Finin, D. Lawrie, P. McNamee, J. Mayfield, D. Oard,N. Gao, N. Peng, Y. Lin, J. MacKin, and T. Dowd.2015. HLTCOE participation in TAC KBP 2015:Cold start and TEDL. In Proceedings of the EighthText Analysis Conference, Gaithersburg, Maryland,November. NIST.

R. Florian, H. Jing, N. Kambhatla, and I. Zitouni.2006. Factorizing complex models: A case studyin mention detection. In Proc. Annual Meetingof the Association for Computational Linguistics(ACL’06), Sydney, Australia, July.

B. Hachey, J. Nothman, and W. Radford. 2014.Cheap and easy entity evaluation. In Proc. ofthe 52nd Annual Meeting of the Association forComputational Linguistics (Vol. 2), pages 464–469.

B. Heinzerling, A. Judea, and M. Strube. 2015. Hitsat tac kbp 2015: Entity discovery and linking, andevent nugget detection. In Proc. Text AnalysisConference (TAC2015).

Y. Hong, D. Lu, D. Yu, X. Pan, X. Wang, Y. Chen,L. Huang, and H. Ji. 2015. Rpi blendertac-kbp2015 system description. In Proc. TextAnalysis Conference (TAC2015).

H. Huang, Y. Cao, X. Huang, H. Ji, and C. Lin.2014. Collective tweet wikification based onsemi-supervised graph regularization. In Proc.the 52nd Annual Meeting of the Association forComputational Linguistics (ACL2014).

H. Ji, D. Westbrook, and R. Grishman. 2005. Usingsemantic relations to refine coreference decisions.In Proc. Human Language Technology Conf./Conf. on Empirical Methods in Natural LanguageProcessing (HLT/EMNLP’05), Vancouver, B.C.,Canada, Oct.

H. Ji, R. Grishman, D. Freitag, M. Blume, J. Wang,S. Khadivi, R. Zens, and H. Ney. 2009. Nameextraction and translation for distillation. Handbookof Natural Language Processing and MachineTranslation: DARPA Global Autonomous LanguageExploitation.

H. Ji, R. Grishman, H.T. Dang, K. Griffitt, and J. Ellis.2010. Overview of the tac 2010 knowledge basepopulation track. In Text Analysis Conf. (TAC) 2010.

H. Ji, R. Grishman, and H. T. Dang. 2011a. Anoverview of the tac2011 knowledge base populationtrack. In Proc. Text Analysis Conf. (TAC’11),Gaithersburg, Maryland, Nov.

H. Ji, R. Grishman, and H.T. Dang. 2011b. Overviewof the tac 2011 knowledge base population track. InText Analysis Conf. (TAC) 2011.

H. Ji, H. T. Dang, J. Nothman, and B. Hachey.2014. Overview of tac-kbp2014 entity discoveryand linking tasks. In Proc. Text Analysis Conference(TAC2014).

C. Lee, Y. Hwang, H. Oh, S. Lim, J. Heo, C. Lee,H. Kim, J. Wang, and M. Jang. 2006. Fine-grainednamed entity recognition using conditional randomfields for question answering. In InformationRetrieval Technology, pages 581–587. Springer.

Q. Li and H. Ji. 2014. Incremental jointextraction of entity mentions and relations. In Proc.the 52nd Annual Meeting of the Association forComputational Linguistics (ACL2014).

Q. Li, H. Ji, Y. Hong, and S. Li. 2014. Constructinginformation networks using one single model. InProc. the 2014 Conference on Empirical Methods onNatural Language Processing (EMNLP2014).

D. Lin, S. Zhao, B. V. Durme, and M. Paca. 2008.Mining parenthetical translations from the webby word alignment. In Proc. Association forComputational Linguistics Conference (ACL2008).

X. Ling, S. Singh, and D. S. Weld. 2014.Context representation for named entity linking. InProc. Pacific Northwest Regional NLP Workshop(NW-NLP 2014).

W. Lu and D. Roth. 2015. Joint mention extractionand classification with mention hypergraphs. InProc. Conference on Empirical Methods in NaturalLanguage Processing (EMNLP2015).

X. Luo. 2005. On coreference resolution performancemetrics. In Proc. HLT/EMNLP, pages 25–32.

X. Pan, T. Cassidy, U. Hermjakob, H. Ji, andK. Knight. 2015. Unsupervised entity linking withabstract meaning representation. In Proc. the 2015Conference of the North American Chapter of theAssociation for Computational Linguistics HumanLanguage Technologies (NAACL-HLT 2015).

L. Ratinov, D. Roth, D. Downey, and M. Anderson.2011. Local and global algorithms fordisambiguation to wikipedia. In Proc. of the AnnualMeeting of the Association of ComputationalLinguistics (ACL).

M. Sammons, H. Peng, Y. Song, S. Upadhyay, C.-T.Tsai, P. Reddy, S. Roy, and D. Roth. 2015. Illinoisccg tac 2015 event nugget, entity discovery andlinking, and slot filler validation systems. In Proc.Text Analytics Conference (TAC2015).

A. Sil and R. Florian. 2015. The ibm systems fortrilingual entity discovery and linking at tac 2015.In Proc. Text Analysis Conference (TAC2015).

H. Wang, J. G. Zheng, X. Ma, P. Fox, and H. Ji.2015. Language and domain independent entitylinking with quantified collective validation. InProc. Conference on Empirical Methods in NaturalLanguage Processing (EMNLP2015).

L. Xiao and D. S. Weld. 2012. Fine-grained entityrecognition. In AAAI.

B. Zhang, H. Huang, X. Pan, H. Ji, K. Knight, Z. Wen,Y. Sun, J. Han, and B. Yener. 2014. Be appropriateand funny: Automatic entity morph encoding.In Proc. Annual Meeting of the Association forComputational Linguistics (ACL2014).

B. Zhang, H. Huang, X. Pan, S. Li, C.-Y. Lin, H. Ji,K. Knight, Z. Wen, Y. Sun, J. Han, and B. Yener.2015. Context-aware entity morph decoding.In Proc. Annual Meeting of the Association forComputational Linguistics (ACL2015).