Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1741–1750Copenhagen, Denmark, September 7–11, 2017. c©2017 Association for Computational Linguistics

KGEval: Accuracy Estimation of Automatically ConstructedKnowledge Graphs

Prakhar OjhaIndian Institute of Science

prakhar@iisc.ac.in

Partha TalukdarIndian Institute of Science

ppt@iisc.ac.in

Abstract

Automatic construction of large knowl-edge graphs (KG) by mining web-scaletext datasets has received considerable at-tention recently. Estimating accuracy ofsuch automatically constructed KGs is achallenging problem due to their size anddiversity and has largely been ignoredin prior research. In this work, we tryto fill this gap by proposing KGEval.KGEval uses coupling constraints to bindfacts and crowdsource those few thatcan infer large parts of the graph. Wedemonstrate that the objective optimizedby KGEval is submodular and NP-hard,allowing guarantees for our approxima-tion algorithm. Through experiments onreal-world datasets, we demonstrate thatKGEval best estimates KG accuracy com-pared to other baselines, while requiringsignificantly lesser number of human eval-uations.

1 Introduction

Automatic construction of Knowledge Graphs(KGs) from Web documents has received signif-icant interest over the last few years, resultingin the development of several large KGs consist-ing of hundreds of predicates (e.g., isCity, sta-diumLocatedInCity(Stadium, City)) and millionsof their instances called beliefs (e.g., (Joe LuisArena, stadiumLocatedInCity, Detroit)). Exam-ples of such KGs include NELL (Mitchell et al.,2015), Knowledge-Vault (Dong et al., 2014) etc.

Due to imperfections in the automatic KG con-struction process, many incorrect beliefs are alsofound in these KGs. Knowing accuracy for eachpredicate in the KG can provide targeted feedbackfor improvement and highlight its strengths from

weaknesses, whereas overall accuracy of a KGcan quantify the effectiveness of its construction-process. Knowing accuracy at predicate-levelgranularity is immensely helpful for Question-Answering (QA) systems that integrate opinionsfrom multiple KGs (Samadi et al., 2015). Forsuch systems, being aware that a particular KG ismore accurate than others in a certain domain, saysports, helps in restricting the search over relevantand accurate subsets of KGs, thereby improvingQA-precision and response time. In comparisonto the large body of recent work focused on con-struction of KGs, the important problem of accu-racy estimation of such large KGs is unexplored –we address this gap in this paper.

True accuracy of a predicate may be estimatedby aggregating human judgments on correctnessof each and every belief in the predicate1. Eventhough crowdsourcing marketplaces such as Ama-zon Mechanical Turk (AMT) provide a convenientway to collect human judgments, accumulatingsuch judgments at the scale of larges KGs is pro-hibitively expensive. We shall refer to the taskof manually classifying a single belief as true orfalse as a Belief Evaluation Task (BET). Thus, thecrucial problem is: How can we select a subsetof beliefs to evaluate which will best estimate thetrue (but unknown) accuracy of KG and its predi-cates?

A naive and popular approach is to evalu-ate randomly sampled subset of beliefs from theKG. Since random sampling ignores relational-couplings present among the beliefs, it usually re-sults in oversampling and poor accuracy estimates.Let us motivate this through an example.

1Note that belief evaluation can not be completely au-tomated and will require human-judgment. If an algorithmcould accurately predict correctness of a belief, then it mayas well be used during KG construction rather than duringevaluation.

1741

Motivating example: We motivate efficient ac-curacy estimation through the KG fragment shownin Figure 1. Here, each belief is an edge-triplein the graph, for example (RedWings, isA, Sport-sTeam). There are six correct and two incorrectbeliefs (the two incident on Taj Mahal), result-ing in an overall accuracy of 75%(= 6/8) whichwe would like to estimate. Additionally, wewould also like to estimate accuracies of the predi-cates: homeStadiumOf, homeCity, stadiumLocate-dInCity, cityInState and isA.

We now demonstrate how evaluation labels ofbeliefs are constrained by each other. Type con-sistency is one such coupling constraint. For in-stance, we know from KG ontology that the home-StadiumOf predicate connects an entity from Sta-dium category to another entity in Sports Team cat-egory. Now, if (Joe Louis Arena, homeStadiumOf,Red Wings) is evaluated to be correct, then fromthese type constraints we can infer that (Joe LouisArena, isA, Stadium) and (Red Wings, isA, SportsTeam) are also correct. Similarly, by evaluating(Taj Mahal, isA, State) as false, we can infer that(Detroit, cityInState, TajMahal) is incorrect.

Additionally, we have Horn-clause couplingconstraints (Mitchell et al., 2015; Lao et al., 2011),such as homeStadiumOf(x, y) ∧ homeCity(y, z)→stadiumLocatedInCity(x, z). By evaluating (RedWings, homeCity, Detroit) and applying this horn-clause to the already evaluated facts mentionedabove, we infer that (Joe Louis Arena, stadium-LocatedInCity, Detroit) is also correct. We ex-plore generalized forms of these constraints inSection 3.1.

Thus, evaluating only three beliefs, and exploit-ing constraints among them, we exactly estimatethe overall true accuracy as 75% and also cover allpredicates. In contrast, the empirical accuracy byrandomly evaluating three beliefs, averaged over 5trials, is 66.7%.

Our contributions in this paper are the follow-ing: (1). Systematic study into the important prob-lem of evaluation of automatically constructedKnowledge Graphs. (2). A novel crowdsourcing-based system KGEval to estimate accuracy oflarge knowledge graphs (KGs) by exploiting de-pendencies among beliefs for more accurate andfaster KG accuracy estimation. (3). Extensiveexperiments on real-world KGs to demonstrateKGEval’s effectiveness and also evaluate its ro-bustness and scalability.

Figure 1: Sample knowledge-graph (KG) fragment that is

consistent but has erroneous beliefs.

All the data and code used in the pa-per are available at http://talukdar.net/mall-lab/KGEval

2 Overview and Problem Statement

2.1 KGEval: OverviewWe try to estimate correctness of as many be-liefs as possible while evaluating only a subset ofthem through crowdsourcing. KGEval achievesthis goal using an iterative algorithm which alter-nates between the following two stages:

• Control Mechanism (Section 3.4): In thisstep, KGEval selects the belief which is to beevaluated next using crowdsourcing.

• Inference Mechanism (Section 3.3): Cou-pling constraints are applied over evaluatedbeliefs to automatically estimate correctnessof additional beliefs.

This iterative process is repeated until there areno more beliefs to be evaluated. Single iterationof KGEval over the KG fragment from Figure 1is shown in Figure 2 where, belief (John LouisArena, homeStadiumOf, Red Wings) is selectedand evaluated by crowdsourcing. Subsequently,the inference mechanism uses type coupling con-straints to infer (JL Arena, isA, Stadium) and (R.Wings, isA, Team) also as true. Next, we formalizethe notations used in this paper.

2.2 Notations and Problem StatementWe are given a KG with n beliefs. Evaluating asingle belief as true or false forms a Belief Evalua-tion Task (BET). Coupling constraints are derivedby determining relationships among BETs, whichwe further discuss in Section 3.1. Notations arealso summarized in Table 1.

1742

Figure 2: Demonstration of one iteration of KGEval. Control mechanism selects a belief whose correctness is evaluated fromcrowd. In the above example, (J.L. Arena, homeStadiumOf, Red Wings) is crowd-evaluated to be true (indicated by tick withdotted square). (Section 2.1 and Section 3).

Symbol DescriptionH = h1, . . . , hn Set of all n Belief Evaluation

Tasks (BETs)c(h) ∈ R+ Cost of labeling h from crowdC = (Ci, θi) Coupling constraints Ci with

weights θi ∈ R+

t(h) ∈ 0, 1 True label of hl(h) ∈ 0, 1 Estimated label of hHi = Dom(Ci) Hi ⊆ H which participate in CiG = (H ∪ C, E) Evaluation Coupling Graph, e ∈

E between Hj and Cj denotesHj ∈ Dom(Cj) .

Q ⊆ H BETs evaluated using crowdI(G,Q) ⊆ H Inferable set for evidenceQ:Φ(Q) =

1|Q|∑h∈Q t(h)

True accuracy of evaluatedBETsQ

Table 1: Summary of notations used (Section 2.2).

Inference algorithm helps us work out evalua-tion labels of other BETs using constraints C. Fora set of already evaluated BETsQ ⊆ H, we defineinferable set I(G,Q) ⊆ H as BETs whose evalu-ation labels can be deduced by the inference algo-rithm. We calculate the average true accuracy of agiven set of evaluated BETs Q ⊆ I(G,Q) ⊆ Hby Φ(Q) = 1

|Q|∑

h∈Q t(h).KGEval aims to sample and crowdsource a BET

set Q with the largest inferable set, and solves theoptimization problem:

arg maxQ⊆H

∣∣I(G,Q)∣∣ (1)

3 KGEval: Method Details

In this section, we describe various components ofKGEval.

3.1 Coupling ConstraintsThe evaluation labels of beliefs are often depen-dent on each other due to rich relational struc-

ture of KGs. In this work, we derive couplingconstraints C from the KG ontology and link-prediction algorithms, such as PRA (Lao et al.,2011) over NELL and AMIE (Galarraga et al.,2013) over Yago. These rules are jointly learnedover entire KG with millions of facts and are as-sumed true.

We use conjunction-form first-order-logic rulesand refer to them as Horn clauses. Examples of afew such coupling constraints are shown below.

C2: (x, homeStadiumOf, y)→ (y, isA, sportsTeam)C5: (x, homeStadiumOf, y) ∧ (y, homeCity, z)→

(x, stadiumLocatedInCity, z)

Each coupling constraint Ci operates over Hi ⊆H to the left of its arrow and infers label ofthe BET on the right of its arrow. C2 enforcestype consistency and C5 is an instance of PRApath. These constraints have also been success-fully employed earlier during knowledge extrac-tion (Mitchell et al., 2015) and integration (Pujaraet al., 2013). Note that the constraints are direc-tional and inference propagates in forward direc-tion.

3.2 Evaluation Coupling Graph (ECG)

To combine all beliefs and constraints at a com-mon place, for all H and C, we construct a graphwith two types of nodes: (1) a node for each BETh ∈ H, and (2) a node for each constraint Ci ∈ C.Each Ci node is connected to all h nodes thatparticipate in it. We call this graph the Evalua-tion Coupling Graph (ECG), represented as G =(H ∪ C, E) with set of edges E = (Ci, h) | h ∈Dom(Ci) ∀Ci ∈ C. Note that ECG is a bipartitefactor graph (Kschischang et al., 2001) with h asvariable-nodes and Ci as factor-nodes.

1743

Figure 3: Evaluation Coupling Graph (ECG) constructed

for example in Figure 1. (Section 3.2)

Figure 3 shows ECG constructed out of the mo-tivating example in Figure 1 with |C| = 8 andseparate nodes for each of the edges (beliefs orBETs) in KG. We pose the KG evaluation prob-lem as classification of BET nodes in the ECG byallotting them a label of 1 or 0 to represent true orfalse respectively.

3.3 Inference Mechanism

Inference mechanism helps propagate true/falselabels of evaluated beliefs to other non-evaluatedbeliefs using available coupling constraints (Bragget al., 2013). We use Probabilistic Soft Logic(PSL), (Broecheler et al., 2010) as our infer-ence engine.Below we briefly describe the inter-nal workings of our inference engine for accuracyestimation problem.

Potential function ψj is defined for each Cj us-ing Lukaseiwicz t-norm and it depicts how satis-factorily constraint Cj is satisfied. For example, C5

mentioned earlier is transformed from first-orderlogical form to a real valued number by

ψj(C5) =(

max0, hx + hy − 1− hw)2 (2)

where C5 = hx ∧ hy → hw, where hx denotes theevaluation score∈ [0, 1] associated with the BETs.

The probability distribution over label assign-ment is so structured such that labels which sat-isfy more coupling constraints are more proba-ble. Probability of any label assignment Ω

(H) ∈0, 1|H| over BETs in G is given by

P(Ω(H)) =

1Z

exp[−|C|∑j=1

θjψj

(Cj)] (3)

where Z is the normalizing constant and ψj cor-responds to potential function acting over BETsh ∈ Dom(Cj). Final assignment of Ω(H)PSL ∈1, 0|H| labels is obtained by solving the maxi-mum a-posteriori (MAP) optimization problem

Ω(H)

PSL= arg max

Ω(H)P(

Ω(H))

We denote by MPSL(h, γ) ∈ [0, 1] the PSL-estimated score for label γ ∈ 1, 0 on BET h inthe optimization above.Inferable Set using PSL: We define estimated la-bel for each BET h as shown below.

l(h) =

1 if MPSL(h, 1) ≥ τ0 if MPSL(h, 0) ≥ τ∅ otherwise

where threshold τ is system hyper-parameter. In-ferable set is composed of BETs for which infer-ence algorithm (PSL) confidently propagates la-bels.

I(G,Q) = h ∈ H | l(h) 6= ∅Note that two BET nodes from ECG can interactwith varying strengths through different constraintnodes; this multi-relational structure requires softprobabilistic propagation.

3.4 Control MechanismControl mechanism selects the BET to be crowd-evaluated at every iteration. We first present thefollowing two theorems involving KGEval’s opti-mization in Equation (1). Please refer Appendixfor proofs of both theorems.Theorem 1. [Submodularity] The function op-timized by KGEval (Equation (1)) using thePSL-based inference mechanism is submodular(Lovasz, 1983).

The proof follows from the fact that all pairs ofBETs satisfy the regularity condition (Jegelka andBilmes, 2011; Kolmogorov and Zabih, 2004), fur-ther used by a proven conjecture (Mossel andRoch, 2007).Theorem 2. [NP-Hardness] The problem of se-lecting optimal solution in KGEval’s optimization(Equation (1)) is NP-Hard.

Proof follows by reducing NP-complete Set-coverProblem (SCP) to selecting Q which coversI(G,Q).

Justification for Greedy Strategy: From The-orem 1 and 2, we observe that the function op-timized by KGEval is NP-hard and submodular.

1744

Algorithm 1 KGEval: Accuracy Estimation ofKnowledge Graphs

Require: H: BETs, C: coupling constraints, B:assigned budget, S: seed set, c(h): BET cost

1: G = BUILDECG(H, C)2: Br = B3: Q0 = S, t = 14: repeat5: h∗ = arg maxh∈H−Q |I(G,Qt−1 ∪ h)|6: CROWDEVALUATE(h∗)7: RUNINFERENCE(Qt−1 ∪ h∗)8: Qt = I(G,Qt−1 ∪ h∗)9: Br = Br − c(h∗)

10: Q = Q∪Qt

11: if Q ≡ H then12: EXIT

13: end if14: Acct = 1

|Q|∑

h∈Q l(h)15: t = t+ 116: until CONVERGENCE

17: return Acct

Results from (Nemhauser et al., 1978) prove thatgreedy hill-climbing algorithms solve such maxi-mization problem within an approximation factorof (1−1/e) ≈ 63% of the optimal solution. Hencewe iteratively select the next BET which gives thegreatest increase in size of inferable set.

We acknowledge the importance of crowd-sourcing aspects such as label-aggregation,worker’s quality estimation etc. Appendix A.1presents a mechanism to handle noisy crowdworkers under limited budget.

3.5 Bringing it All TogetherAlgorithm 1 presents KGEval. In Lines 1-3, webuild the Evaluation Coupling Graph G = (H ∪C, E) and use the labels of seed set S to initial-ize G. In lines 4-16, we repetitively run our in-ference mechanism, until either the accuracy esti-mates have converged, or all the BETs are covered.In each iteration, the BET with the largest infer-able set is identified and evaluated using crowd-sourcing (Lines 5-6). The new inferable set Qt isestimated. These automatically annotated nodesare added to Q (Lines 7-10).Convergence: In this paper, we define conver-gence whenever the variance of sequence of accu-racy estimates [ Acct−k, . . ., Acct−1, Acct] is lessthan α. We set k = 9 and α = 0.002 for ourexperiments.

4 Experiments

To assess the effectiveness of KGEval, we ask thefollowing questions: (1).How effective is KGEvalin estimating KG accuracy, both at predicate-leveland at overall KG-level? (Section 4.3). (2). Whatis the importance of coupling constraints on itsperformance? (Section 4.4). (3). And lastly, howrobust is KGEval to estimating accuracy of KGswith varying quality? (Section 4.5).

4.1 Model Description

Evaluation set HN HY#BETs 1860 1386

#Constraints |CN | = 130 |CY | = 28#Predicates 18 16Gold Acc. 91.34% 99.20%

Table 2: Details of BET subsets used for accuracy evalua-tion. (Section 4.1.2).

4.1.1 SetupDatasets: For experiments, we consider twoKGs: NELL and Yago2. From NELL (NELL),we choose a sub-graph of sports related be-liefs NELL-sports, mostly pertaining to ath-letes, coaches, teams, leagues, stadiums etc.We construct coupling constraints set CN us-ing top-ranked PRA inference rules for availablepredicate-relations (Lao et al., 2011). The confi-dence score returned by PRA are used as weightsθi. We use NELL-ontology’s predicate-signaturesto get information for type constraints. Pleasenote that PSL is capable of handling weightedconstraints and also learn their weights (relativeimportance). So, it is not critical to provideabsolutely correct constraints. We also selectYAGO2-sample (YAGO) , which unlike NELL-sports, is not domain specific. We use AMIE hornclauses (Galarraga et al., 2013) to construct multi-relational coupling constraints CY. For each Ci,the score returned by AMIE is used as rule weightθi. Table 2 reports the statistics of datasets used,their true accuracy and number of coupling con-straints. Obtaining gold-labels for millions of factsis non-trivial and expensive as crowdsourcing overfull KG incurs significant cost.

Size of evaluation set: NELL-sport consists of23, 422 beliefs with 13, 290 unique entities and53 unique predicates. Whereas YAGO-sample has31, 720 beliefs, with unique 32, 103 entities and17 predicates. In order to calculate accuracy, we

1745

require gold evaluation of all beliefs in the evalu-ation set. Since obtaining gold evaluation of theentire (or large subsets of) NELL and Yago2 KGswill be prohibitively expensive, we take subset ofthese KGs for evaluation. (KGEval) consists ofdatasets used, their crowdsourced labels, couplingconstraints and code for inference/control.

Initialization: Algorithm 1 requires initial seedset S which we generate by randomly evaluating|S| = 50 beliefs from H. To maintain fairness,all baselines start from S. For asserting true (orfalse) value for beliefs, we set a high soft labelconfidence threshold at τ = 0.8 (see Section 3.3).

4.1.2 Crowdsourcing of BETsTo compare KGEval predictions against humanevaluations, we evaluate all BETs HN ∪ HY on AMT. For the ease of workers, we translateeach entity-relation-entity belief into human read-able format before posting to crowd.

We published BETs on AMT under ‘classifi-cation project’ category. We hired AMT recog-nized master workers for high quality labels andpaid $0.01 per BET. To compare between ‘mas-ter’ and ‘noisy’ workers, we correlated their la-bels individually to expert labels on random subsetand observed that master workers were better cor-related (93%) as compared to three non-masters(89%). Consequently we consider votes of mas-ter workers for HN ∪HY as gold labels, whichwe would like our inference algorithm to be ableto predict. As the average turnaround time forAMT tasks runs into a few minutes (Dupuis et al.,2013), KGEval is effectively real-time within suchturnaround time range.

4.1.3 Performance Evaluation MetricsPerformance of various methods are evaluated us-ing the following two metrics. To capture accuracyat the predicate level, we define ∆predicate as theaverage of difference between gold and estimatedaccuracy of each of the R predicates in KG.

∆predicate =1

|R|

( ∑∀r∈R

∣∣∣Φ(Hr)− 1

|Hr|∑∀h∈Hr

l(h)∣∣∣)

We define ∆overall as the difference betweengold and estimated accuracy over the entire evalu-ation set.

∆overall =

∣∣∣∣Φ(H)− 1

|H|∑∀h∈H

l(h)

∣∣∣∣

Above, Φ(H) is the overall gold accuracy,Φ(Hr) is the gold accuracy of predicate r andl(h) is the label assigned by the currently evalu-ated method. ∆overall treats entire KG as a singlebag of BETs whereas ∆predicate segregates beliefsbased on their type of predicate-relation. For bothmetrics, lower is better.

4.2 Baseline MethodsSince accuracy estimation of large multi-relationalKGs is a relatively unexplored problem, there areno well established baselines for this task (apartfrom random sampling). We present below thebaselines which we compared against KGEval.Random: Randomly sample a BET h ∈ H with-out replacement and crowdsource for its correct-ness. Selection of every subsequent BET is inde-pendent of previous selections.Max-Degree: Sort the BETs in decreasing or-der of their degrees in ECG and select them fromtop for evaluation; this method favors selection ofmore centrally connected BETs first.Independent Cascade: This method is based oncontagion transmission model where nodes onlyinfect their immediate neighbors (Kempe et al.,2003). At every time iteration t, we choose a BETwhich is not evaluated yet, crowdsource for its la-bel and let it propagate its evaluation label in ECG.KGEval: Method proposed in Algorithm 1.

4.3 Effectiveness of KGEvalExperimental results of all methods comparing∆overall and ∆predicate at convergence, are pre-sented in Table 3. We observe that KGEval is ableto achieve the best estimate across both datasetsand metrics. Due to the significant positive biasin HY (see Table 2), all methods do fairly well asper ∆overall on this dataset, even though KGEvalstill outperforms others. Also, KGEval is able toestimate KG accuracy most closely while utiliz-ing least number of crowd-evaluated queries. Thisclearly demonstrates KGEval’s effectiveness.

Nodes in coupling graph with higher degreesare those which participate in large number of con-straints. In real KGs, such facts tend to be correctas they interact with several other facts. Hence,MaxDegree overestimates the accuracy by propa-gating True label. In contrast, Random samplesTrue and False labels in unbiased fashion.Predicate-level Analysis: Here, we consider thetop two baselines from Table 3, viz., Random and

1746

NELL sports dataset (HN )Method ∆predicate ∆overall # Queries

(%) (%)Random 4.9 1.3 623Max-Deg 7.7 2.9 1370Ind-Casc 9.8 0.8 232KGEval 3.6 0.5 140

Yago dataset (HY )Random 1.3 0.3 513Max-Deg 1.7 0.5 550Ind-Casc 1.1 0.7 649KGEval 0.7 0.1 204

Table 3: ∆predicate(%) and ∆overall(%) estimates (loweris better) of various methods with number of crowd-evaluatedqueries (BET evaluations) to reach the ∆overall convergedestimate. (See Section 4.3)

wor

ksF

or

team

Pla

ysIn

city

spor

tUse

sSta

diu

m

team

Hom

eSta

diu

m

ath

lete

Hom

eSta

diu

m

coac

hes

Tea

m

ath

lete

Pla

ysF

orT

eam

stad

ium

Loc

ated

InC

ity

team

Pla

ysIn

Lea

gue

ath

lete

Pla

ysS

por

t

ath

lete

Led

Sp

orts

Tea

m

ath

lete

Pla

ysIn

Lea

gue

spor

tsG

ameL

oser

ath

lete

Coa

ch

Ave

rage

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

1−

[∆overall] predicate

Random

KGEval

Figure 4: Comparing (1− [∆overall]predicate) of individ-ual predicates (higher is better) in HN between KGEval andRandom, the two top performing systems in Table 3. (seeSection 4.3)

KGEval, and compare performance on the HN

dataset. We use (1 − [∆overall]predicate) as themetric, which means ∆overall computed over indi-vidual predicates. Here, we are interested in eval-uating how well the two methods have estimatedper-predicate accuracy when KGEval’s ∆overall

has converged. Comparison of per-predicate per-formances of the two methods is shown in Fig-ure 4. Observe that KGEval significantly outper-forms Random baseline. Its advantage lies in ex-ploiting the coupling constraints among beliefs,where evaluating a belief from certain predicatehelps infer beliefs from other predicates.

Constraint Set Iterations to ∆overall(%)Convergence

C 140 0.5C − Cb3 259 0.9C − Cb3 − Cb2 335 1.1

Table 4: Performance of KGEval with ablated constraintsets. Additional constraints help in better estimation withlesser iterations.(see Section 4.4)

4.4 Importance of Coupling Constraints

This paper is largely motivated by the thesis – ex-ploiting richer relational couplings among BETsmay result in faster and more accurate evalua-tions. To evaluate this thesis, we successively ab-lated Horn clause coupling constraints of body-length 2 and 3 from CN .

We observe that with the full (non-ablated) con-straint set CN , KGEval takes least number ofcrowd evaluations of BETs to convergence, whileproviding best accuracy estimate. Whereas withablated constraint sets, KGEval takes up to 2.4xmore crowd-evaluation queries for convergence;thus validating our thesis.

4.5 Additional Experiments

Other Baselines along with Inference: In or-der to evaluate how Random and Max-degree per-form in conjunction with inference mechanism,we replaced KGEval’s greedy control mechanismin Line 5 of Algorithm 1 with these two con-trol mechanisms. In our experiments, we ob-served that both Random+inference and Max-degree+inference are able to estimate accuracymore accurately than their control-only variants.Secondly, even though the accuracies estimatedby Random+inference and Max-degree+inferencewere comparable to that of KGEval, they requiredlarger number of crowd-evaluation queries – 1.2xand 1.35x more, respectively. This shows effec-tiveness of greedy mechanism.

Rate of Coverage: In case of large KGs withscarce budget, it is imperative to have a mecha-nism which covers greater parts of KG with lessernumber of crowdsource queries. Figure 5 showsthe fraction of total beliefs whose evaluations wereautomatically inferred by different methods as afunction of number of crowd-evaluated beliefs.We observe that KGEval infers evaluation for thelargest number of BETs at each supervision level.

Robustness to Noise: In order to test ro-bustness of the methods in estimating accuracies

1747

50+5 50+10 50+100No. of BETs Crowd-Evaluated

0.0

0.2

0.4

0.6

0.8

1.0

Fra

ctio

nof

Tot

alB

elie

fsE

valu

ated

Random

Max-Degree

Independent Cascade

KGEval

Figure 5: Fraction of total beliefs whose evaluation whereautomatically inferred by different methods for varying num-ber of crowd-evaluated queries (x-axis) inHN .

NELL sports + 5% noise (HN5)Method ∆overall (%) # QueriesRandom 1.8 563Max-Degree 4.2 1249Ind-Cascade 1.2 370KGEval 0.8 106

NELL sports + 10% noise (HN10)Random 1.8 728Max-Degree 6.2 1501Ind-Cascade 1.2 406KGEval 0.2 115

Table 5: Accuracy estimate (higher is better) over entireKG by various baselines in the presence of noise.

of KGs with different gold accuracies, we arti-ficially added noise to HN by flipping a fixedfraction of edges, otherwise following the sameevaluation procedure as in Section 3.5. We ana-lyze ∆overall (and not ∆predicate) because flippingedges in KG distorts predicate-relations dependen-cies and present in Table 5. We evaluated all themethods and observed that while performance ofother methods degraded significantly with dimin-ishing KG quality (more noise), KGEval was sig-nificantly robust to noise.

Scalability comparisons with MLN: MarkovLogic Networks (Richardson and Domingos,2006) can serve as a candidate for Inference Mech-anism. We compared the runtime performance ofKGEval with PSL and MLN as inference engines.While PSL took 320 seconds to complete one iter-ation, the MLN implementation (PyMLN) couldnot finish grounding the rules even after 7 hours.This justifies our choice of PSL as the inferenceengine for KGEval.

5 Related Work

Even though Knowledge Graph (KG) construc-tion is an active area of research, we are notaware of any previous research which systemati-

cally studies the important problem of estimatingaccuracy of such automatically constructed KGs.Random sampling has traditionally been the mostpreferred way for large-scale KG accuracy estima-tion (Mitchell et al., 2015).

Traditional crowdsourcing research has typi-cally considered atomic allocation of tasks wherethe requester posts them independently. KGEvaloperates in a rather novel crowdsourcing set-ting as it exploits dependencies among its tasks(BETs or belief evaluations). Our notion of inter-dependence (coupling constraints) among tasksis more general and different than related ideasexplored in the crowdsourcing literature before(Kolobov et al., 2013; Bragg et al., 2013; Sunet al., 2015). Even though coupling constraintshave been used for KG construction (Nakasholeet al., 2011; Galarraga et al., 2013; Mitchell et al.,2015), they have so far not been exploited for KGevaluation. We address this gap in this paper.

The task of knowledge corroboration (Kasneciet al., 2010) proposes probabilistic model to uti-lize a fixed set of basic first-order logic rules forlabel propagation and is closely aligned with ourmotivations. However, unlike KGEval, it does nottry to reduce the number of queries to crowdsourceor maximize coverage.

6 Conclusion

In this paper, we have initiated a systematic studyinto the important problem of evaluation of auto-matically constructed Knowledge Graphs. In or-der to address this challenge, we have proposedKGEval, an instance of a novel crowdsourcingparadigm where dependencies among tasks pre-sented to humans (BETs) are exploited. To thebest of our knowledge, this is the first method ofits kind. We demonstrated that the objective opti-mized by KGEval is in fact NP-Hard and submod-ular, and hence allows for the application of sim-ple greedy algorithms with guarantees. Throughextensive experiments on real datasets, we demon-strated effectiveness of KGEval. We hope to ex-tend KGEval to incorporate varying evaluationcost, and also explore more sophisticated evalu-ation aggregation.

Acknowledgments

This research has been supported by MHRD(Govt. of India) and in part by a gift from Google.

1748

A AppendixSubmodularity: A real valued function f , which actsover subsets of any finite set H, is said to be submodularif ∀R,S ⊂ H it fulfills

f(R) + f(S) ≥ f(R ∪ S) + f(R ∩ S).

We call potential functionψ as pairwise regular if for all pairsof BETs p, q ∈ H it satisfies

Proof. (for Theorem 1) The additional utility, in terms of la-bel inference, obtained by adding a BET to larger set is lesserthan adding it to any smaller subset. By construction, any twoBETs which share a common factor node Cj are encouragedto have similar labels in G.

Potential functionsψj of Equation (3) satisfy pairwise reg-ularity property i.e., for all BETs p, q ∈ H

ψ(0, 1) + ψ(1, 0) ≥ ψ(0, 0) + ψ(1, 1) (4)

where 1, 0 represent true/false. Equivalence of submodu-lar and regular properties are established (Kolmogorov andZabih, 2004; Jegelka and Bilmes, 2011). Using non-negativesummation property (Lovasz, 1983),

∑j∈C θjψj is submod-

ular for positive weights θj ≥ 0.We consider a BET h to be confidently inferred when the

soft score of its label assignment in I(G,Q) is greater thanthreshold τh ∈ [0, 1]. From above we know that P(l(h)|Q) issubmodular with respect to fixed initial setQ. Although maxor min of submodular functions are not submodular in gen-eral, but (Kempe et al., 2003) conjectured that global func-tion of Equation (1) is submodular if local threshold functionP(h|Q) ≥ τh respected submodularity, which holds good inour case of Equation (3). This conjecture was further provedin (Mossel and Roch, 2007) and thus making our global opti-mization function of Equation (1) submodular.

Proof. (for Theorem 2) We reduce KGEval to NP-completeSet-cover Problem (SCP) so as to select Q which coversI(G,Q). For the proof to remain consistent with earlier nota-tions, we define SCP by collection of subsets I1, I2, . . . , Imfrom set H = h1, h2, . . . , hn and we want to deter-mine if there exist k subsets whose union equals H. Wedefine a bipartite graph with m + n nodes correspondingto Ii’s and hj’s respectively and construct edge (Ii, hj) ifhj ∈ Ii. We need to find a set Q, with cardinality k, suchthat |I(G,Q)| ≥ n+ k.

Choosing our BET-set Q from SCP solution and furtherinferring evaluations of other remaining BETs using PSL willsolve the problem in hand.

A.1 Noisy Crowd Workers and BudgetHere, we provide a scheme to allot crowd workers so as toremain within specified budget and upper bound total erroron accuracy estimate. We have not integrated this mechanismwith Algorithm 1 to maintain its simplicity.

We resort to majority voting in our analysis and assumethat crowd workers are not adversarial. So expectation overresponses rh(u) for a task h with respect to multiple workersu is close to its true label t(h) (Tran-Thanh et al., 2013), i.e.,∣∣∣∣Eu∼D(u,h)[rh(u)]− t(h)

∣∣∣∣ ≤ 1

2(5)

where D is joint probability distribution of workers u andtasks h.

Our key idea is that we want to be more confident aboutBETs h with larger inferable set (as they impact largerparts of KG) and hence allocate them more budget to post

to more workers. We determine the number of workerswh1 , . . . , whn for each task such that ht with larger in-ference set have higher wht . For total budget B, we allocate

wht =⌊B it (1−γ)

c imax

⌋where it denotes the cardinality of inferable set I(G,Q∪ht),c the cost of querying crowd worker, imax the size of largestinferable set and γ ∈ [0, 1] constant.

This allocation mechanism easily integrates with Algo-rithm 1; in (Line 8) we determine size of inferable set it =|Qt| for task h and allocatewh crowd workers. Budget deple-tion (Line 9) is modified to Br = Br −whc(h). The follow-ing theorem bounds the error with such allocation scheme.

Theorem 3. [Error bounds] The allocation scheme of re-dundantly posing ht to wht workers does not exceed the totalbudget B and its expected estimation error is upper boundedby e−O(it), keeping other parameters fixed. The expected es-timation error over all tasks is upper bounded by e−O(B).

Proof. Let γ ∈ [0, 1] control the reduction in size of infer-able set by it+1 = γ it. By allocating wht redundant work-ers for task ht, ∀t ∈ 1 · · ·n with size of inferable set it,we incur total cost of

n∑t=1

c wht =

n∑t=1

B it (1− γ)

c imax· c

=

(n∑t=1

it

)·(B (1− γ)

imax

)=

(imax (1− γT )

(1− γ)

)·(B (1− γ)

imax

)≤ B

Note that the above geometric approximation helps in esti-mating summation

∑nt=1 it at iteration t ≤ n.

Error Bounds: Here we show that the expected error ofestimating of ht for any time t decreases exponentially in thesize of inferable set it. We use majority voting to aggregatewht worker responses for ht, denoted by rht ∈ 0, 1

rht =

⌊1

wht

wht∑k=1

rht(uk)− 1

2

⌋+ 1 (6)

where rht(uk) is the response by kth worker for ht. Theerror from aggregated response can be given by ∆(ht) =|rht− t(ht)|, where t(ht) is its true label. From Equation (5)and Hoeffding-Azuma bounds over wht i.i.d responses anderror margin εt, we have

∆(ht) = P

∣∣∣∣∣ 1

wht

wht∑k=1

rht(uk)− E(rh(u))

∣∣∣∣∣ ≥ εt

= 2 exp

(−2

B it (1− γ)

c imaxε2t

)For fixed budget B and given error margin εt, we have∆(ht) = e−O(it). Summing up over all tasks t, by unionbounds we get the total expected error from absolute truth as∆(B) =

∑nt=1 ∆(ht).

∆(B) ≤n∑t=1

2 exp

(−2

B it (1− γ)

c imaxε2t

)≤ n · 2 exp

(−2

B imin (1− γ)

c imaxε2min

)The accuracy estimation error will decay exponentially withincrease in total budget for fixed parameters.

1749

ReferencesAMT. https://www.mturk.com.

Jonathan Bragg, Daniel S Weld, et al. 2013. Crowd-sourcing multi-label classification for taxonomy cre-ation. In HCOMP.

Matthias Broecheler, Lilyana Mihalkova, and LiseGetoor. 2010. Probabilistic similarity logic. In UAI.

Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, WilkoHorn, Ni Lao, Kevin Murphy, Thomas Strohmann,Shaohua Sun, and Wei Zhang. 2014. Knowledgevault: A web-scale approach to probabilistic knowl-edge fusion. In SIGKDD, pages 601–610.

Marc Dupuis, Barbara Endicott-Popovsky, and RobertCrossler. 2013. An analysis of the use of amazonsmechanical turk for survey research in the cloud. InICCSM2013-Proceedings of the International Con-ference on Cloud Security Management: ICCSM2013, page 10. Academic Conferences Limited.

Luis Antonio Galarraga, Christina Teflioudi, KatjaHose, and Fabian Suchanek. 2013. Amie: associa-tion rule mining under incomplete evidence in onto-logical knowledge bases. In WWW, pages 413–422.

Stefanie Jegelka and Jeff Bilmes. 2011. Submodular-ity beyond submodular energies: coupling edges ingraph cuts. In CVPR, pages 1897–1904.

Gjergji Kasneci, Jurgen Van Gael, Ralf Herbrich, andThore Graepel. 2010. Bayesian knowledge cor-roboration with logical rules and user feedback.In Machine Learning and Knowledge Discovery inDatabases, pages 1–18.

David Kempe, Jon Kleinberg, and Eva Tardos. 2003.Maximizing the spread of influence through a socialnetwork. In SIGKDD.

KGEval. http://talukdar.net/mall-lab/KGEval.

Vladimir Kolmogorov and Ramin Zabih. 2004. Whatenergy functions can be minimized via graph cuts?Pattern Analysis and Machine Intelligence, IEEETransactions on, 26(2):147–159.

Andrey Kolobov, Daniel S Weld, et al. 2013. Jointcrowdsourcing of multiple tasks. In HCOMP.

Frank R Kschischang, Brendan J Frey, and H-ALoeliger. 2001. Factor graphs and the sum-productalgorithm. Information Theory, IEEE Transactionson, 47(2):498–519.

Ni Lao, Tom Mitchell, and William W Cohen. 2011.Random walk inference and learning in a large scaleknowledge base. In EMNLP, pages 529–539.

Laszlo Lovasz. 1983. Submodular functions and con-vexity. In Mathematical Programming The State ofthe Art, pages 235–257. Springer.

T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Bet-teridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel,J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed,N. Nakashole, E. Platanios, A. Ritter, M. Samadi,B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen,A. Saparov, M. Greaves, and J. Welling. 2015.Never-ending learning. In Proceedings of AAAI.

Elchanan Mossel and Sebastien Roch. 2007. On thesubmodularity of influence in social networks. InACM symposium on Theory of computing, pages128–134.

Ndapandula Nakashole, Martin Theobald, and GerhardWeikum. 2011. Scalable knowledge harvesting withhigh precision and high recall. In Proceedings ofWSDM.

NELL. http://rtw.ml.cmu.edu/rtw/resources.

George L Nemhauser, Laurence A Wolsey, and Mar-shall L Fisher. 1978. An analysis of approximationsfor maximizing submodular set functionsi. Mathe-matical Programming, 14(1):265–294.

PSL. http://www.psl.umiacs.umd.edu.

Jay Pujara, Hui Miao, Lise Getoor, and William Cohen.2013. Knowledge graph identification. In The Se-mantic Web–ISWC 2013, pages 542–557. Springer.

PyMLN. http://ias.cs.tum.edu/people/jain/mlns.

Matthew Richardson and Pedro Domingos. 2006.Markov logic networks. Machine learning, 62(1-2):107–136.

Mehdi Samadi, Partha Talukdar, Manuela Veloso, andTom Mitchell. 2015. Askworld: budget-sensitivequery evaluation for knowledge-on-demand. In Pro-ceedings of the 24th International Conference on Ar-tificial Intelligence, pages 837–843. AAAI Press.

Yuyin Sun, Adish Singla, Dieter Fox, and AndreasKrause. 2015. Building hierarchies of concepts viacrowdsourcing. arXiv preprint arXiv:1504.07302.

Long Tran-Thanh, Matteo Venanzi, Alex Rogers, andNicholas R Jennings. 2013. Efficient budget allo-cation with accuracy guarantees for crowdsourcingclassification tasks. In AAMAS.

YAGO. https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga.

1750