Amrapali Zaveri Defense

17th April, 2015! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Leipzig, Germany

Linked Data Quality Assessment and its Application to Societal Progress Measurement

Amrapali Zaveri

1

Faculty of Mathematics and Computer Science!!

Supervisors:!Prof. Dr. Ing. habil. Klaus-Peter Fähnrich, University of Leipzig!

Dr. Jens Lehmann, University of Leipzig! Prof. Dr. Sören Auer, University of Bonn

Outline

2Linked Data Quality Assessment and its Application to Societal Progress Measurement A. Zaveri

Outline

Motivation — Linked Data Quality


Outline


Linked Data Quality Assessment Methodologies


Outline



Use Case Leveraging Data Quality


Outline




Contributions


Outline




Contributions

Future Work


Motivation!

— Linked Data Quality

Linked Data Quality Assessment and its Application to Societal Progress Measurement A. Zaveri 3

Data on the Web

4


Linked Data Quality Assessment and its Application to Societal Progress Measurement A. Zaveri

Data on the Web

5



Data on the Web

5

Accessible



Data on the Web

5

Accessible

Re-usable



Data on the Web

5

Accessible

Re-usable

Understandable



Data on the Web

5

Accessible

Re-usable

Understandable

Discoverable



Linked Data Principles

6




6

Use URIs as names for things.




6


Use HTTP URIs, so that people can look up those names.




6



When someone looks up a URI, provide useful information, using the standards (RDF, RDFS, OWL, SPARQL).




6



When someone looks up a URI, provide useful information, using the standards (RDF, RDFS, OWL, SPARQL).

Include links to other URIs, so that they can discover more things.



Linked Data

7


Linked Data



Linked Data

9


Linked Data

9

What about the quality?


Data Quality

10

Data Quality is defined as:!

“fitness for use”*!

* Juran, J. (1974). The Quality Control Handbook. McGraw-Hill, New York.



Consequences of Poor Quality



*http://www.gartner.com/newsroom/id/501733!#http://www.mckinsey.com/insights/business_technology/open_data_unlocking_innovation_and_performance_with_liquid_information

http://www.gartner.com/newsroom/id/501733

http://www.mckinsey.com/insights/business_technology/open_data_unlocking_innovation_and_performance_with_liquid_information


11

Propagation of errors in integrated datasets







11


Major hindrance in acquiring reliable results







11



Loss of important information







11



Loss of important information

Loss in productivity — Additional costs*#






Data Quality Assessment



Data Quality Assessment

12

How can one assess the quality of data and make this information explicit?!

Which criteria should be assessed?!

Which measures should be used?!

Which methodologies/tools can be utilized?



Main Research Question



Main Research Question

13

How can we exploit Linked Data for a particular use case and ensure good data quality?



Overview

14

Systematic!literature!

review

Linked Data Quality Assessment !Methodologies Evaluation

User-driven Crowdsourcing

Semi-!automated

Use case!leveraging!

quality


Overview

15


review



Semi-!automated


quality


Current State


Systematic Literature Review

Current State

16

Lack of unified descriptions for data quality dimensions and metrics for Linked Data



Current State

16


Lack of use-case-driven data quality assessment methodologies for Linked Data



Current State

16


Lack of use-case-driven data quality assessment methodologies for Linked Data

Lack of quality assessment of datasets before utilisation in particular use cases



17

Research Questions



17

RQ1 What are the existing approaches to assess the quality of Linked Data employing a conceptual framework integrating prior approaches?!

RQ1.1 What are the data quality problems that each approach assesses?!RQ1.2 Which are the data quality dimensions and metrics supported by the proposed approaches?!RQ1.3 Which tools already exist to assess the quality of Linked Data?

Research Questions



Qualitative Analysis

18

Quality assessment methodologies for Linked Data: A Survey. Amrapali Zaveri, Anisa Rula, Andrea Maurino, Ricardo Pietrobon, Jens Lehmann and Sören Auer. Semantic Web Journal 2015.




18

30 core articles





18

30 core articles18 dimensions - definitions





18

30 core articles18 dimensions - definitions69 metrics





18

30 core articles18 dimensions - definitions69 metrics12 tools compared using 8 attributes




Dimensions



*specific for Linked Data

Dimensions

Relevancy Conciseness

Timeliness

Rep.-Conciseness

Interoperability

Consistency

Interpretability

Understandability

Versatility*

Availability

Performance* Interlinking*

SyntacticValidity

Representation

ContextualIntrinsic

Accessibility

Trustworthiness

Two dimensionsare related

Licensing*

Semantic Accuracy

Completeness

Security*

Dim1 Dim2



*specific for Linked Data

Metrics

20

Linked Data Quality MetricsDimension Metric Description QN/QL*

Completeness Schema completeness No. of classes and properties / !total no. of classes and properties QN

Interlinking Detection of good quality interlinks

(i) detection of (a) interlinking degree, (b) clustering coefficient, (c) centrality, (d) open sameAs chains and (e) description richness through sameAs by using network measures, (ii) via crowdsourcing

QN

Timeliness Freshness of datasets Max{0, 1 − currency / volatility} QN

Trustworthiness Trustworthiness of information provider

indicating the level of trust for the publisher on a scale of 1−9 QL

*QN - Quantitative Metric ; *QL - Qualitative Metric


Tools

21

Trellis TrustBOT tSPARQL WIQA ProLOD Flemming

Availablility - - ✔ - - ✔

Licensing Open-source - GPL v3 Apache v2 - -

Automation Semi-automated

Semi-automated

Semi-automated

Semi-automated

Semi-automated

Semi-automated

Collaboration Yes No No No No No

Customizability ✔ ✔ ✔ ✔ ✔ ✔

Scalability - No Yes - - No

Usability 2 4 4 2 2 3

Maintainance 2005 2003 2012 2006 2010 2010



Tools

22

LinkQA Sieve RDFUnit DaCura TripleCheckMate LiQuate

Availablility ✔ ✔ ✔ - ✔ ✔

Licensing Open-source

Apache Apache - Apache -

Automation Automated Semi-automated

Semi-automated

Semi-automated

Semi-automated Semi-automated

Collaboration No No No Yes yes No

Customizability No✔ ✔ ✔ ✔

No

Scalability Yes Yes Yes No Yes No

Usability 2 4 3 1 5 1

Maintainance 2011 2012 2014 2013 2013 2013



Problems in Current Approaches




23

Not catered to the use case




23


Results difficult to interpret




23



Do not report the root cause of the quality issues




23




Require considerable amount of configuration




23




Require considerable amount of configuration

Do not allow user to choose input dataset



Overview

24


review



Semi-!automated


quality


Research Questions

25

User-Driven Quality Assessment


Research Questions

25

RQ2 How can we assess the quality of Linked Data using a user-driven methodology?!

RQ2.1 How feasible is it to employ Linked Data experts to assess the quality issues of LD?!RQ2.2 How feasible is it to use a combination of user-driven and semi-automated methodology to assess the quality of LD?



Methodology

26

Resource Selection

[Per Class] [Manual]

[Random]

Resource

Evaluation mode selection

Resource Evaluation

[Manual]

Triples

[Semi-automatic] [Automatic]

List of invalid facts

Data QualityImprovement

Pre-selection of triples

Patch Ontology



Methodology

26

Resource Selection

[Per Class] [Manual]

[Random]

Resource

Evaluation mode selection

Resource Evaluation

[Manual]

Triples

[Semi-automatic] [Automatic]

List of invalid facts

Data QualityImprovement

Pre-selection of triples

Patch Ontology



Manual!Semi-automated!

!

!

!

!

Manual — Phase I

27

Linked Data Quality Problem TaxonomyDimensions Category

AccuracyTriple incorrectly extracted!Datatype problems!Implicit relationships between attributesRelevancy Irrelevant information extracted

Representational consistency Representation of number values

Interlinking External linksInterlinks with other datasets



Manual — Phase II

28



Manual — Phase II

28

Invited Linked Data experts!

Triple-based evaluation!

Contest-based - 3 weeks



Phase II — TripleCheckMate

29


https://github.com/AKSW/TripleCheckMate


Choose a resource

https://github.com/AKSW/TripleCheckMate


30



Identify erroneous triples


31


Map to the quality problem taxonomy

Manual — Results

32

Total no. of users 58

Total no. of distinct resources evaluated 521Total no. of distinct incorrect triples 2928% of triples affected 11.93%

Resource-based inter-rater agreement (Cohen’s kappa) 0.34

Total no. of triples evaluated for correctness 700

% of triples evaluated incorrectly 19%



Semi-automated — Step 1

33


*Lehmann, J. (2009). DL-Learner: learning concepts in description logics. Journal of Machine Learning Research, 10:2639–2642.!



33

Generate schema axioms for properties via DL-Learner*





33


Functionality





33


FunctionalityInverse functionality





33


FunctionalityInverse functionalityAsymmetric





33


FunctionalityInverse functionalityAsymmetricIrreflexivity





33






Example:


33






Example:Domain: Formula One Racer


33






Example:Domain: Formula One RacerRange: Grand Prix


33






Example:Domain: Formula One RacerRange: Grand PrixOnly 1 first win of each Formula One Racer (Functional)


34




34

Manual evaluation of generated axioms




34

Manual evaluation of generated axioms100 random axioms per type




34

Manual evaluation of generated axioms100 random axioms per typeOnly those axioms where at least one violation can be found




34

Manual evaluation of generated axioms100 random axioms per typeOnly those axioms where at least one violation can be foundAlso taking target context into account



Semi-automated — Results

35


Inverse!functionality

Functionality

Asymmetry

Irreflexivity


Summary

36



Summary

36

Quality analysis of over 500 resources



Summary

36

Quality analysis of over 500 resources12% error detected



Summary

36

Quality analysis of over 500 resources12% error detectedLinked Data experts performed quality analysis but evaluated correct triples as errors



Summary

36

Quality analysis of over 500 resources12% error detectedLinked Data experts performed quality analysis but evaluated correct triples as errors 75% functionality violations of property characteristics detected but required manual verification



Overview

37


review



Semi-!automated


quality


Research Questions

38

Crowdsourcing Linked Data Quality Assessment


Research Questions

38

RQ2.3 Is it possible to detect quality issues in LD datasets via crowdsourcing mechanisms?!

RQ2.4 What type of crowd is most suitable for each type of quality issue?!

RQ2.5 Which types of assessment errors are made by lay users and experts?



Concepts

39


- Crowdsourcing Linked Data quality assessment. Maribel Acosta, Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Sören Auer and Jens Lehmann. ISWC 2013.


- Detecting Linked Data Quality Issues via Crowdsourcing: A DBpedia Study. Maribel Acosta, Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Fabian Flöck and Jens Lehmann. SWJ (Submitted) 2015.

Concepts

39

AMT - Amazon Mechanial Turk





Concepts

39


HITs - Human Intelligent Tasks/microtasks





Concepts

39



MTurk Workers - monetary reward for each HIT





Concepts

39



MTurk Workers - monetary reward for each HIT

Find-Fix-Verify phases





Methodology

40

Resource

[Manual]

[Any]

Resource selection

Evaluation of resource’s

triples

Selection of quality issues

[Incorrect triples]

[Yes]

[No]

List of incorrect triples classified by quality issue

(Find stage) LD Experts in contest

HIT generation

(Verify stage) Workers in paid microtasks

Accept HIT

Assess triple according to

the given quality issue

Submit HIT

[Per Class]

[Correct]

[Incorrect]

[Data doesn’t make sense] [I don’t know]

[More triples to assess]

[No]

[Yes]

Experts Workers



Quality Issues Types

41



41

Incorrect/incomplete object value



41

Incorrect/incomplete object valuedbpedia:Oreye! !dbpedia-owl:postalCode! !“4360”!@en

Incorrect datatypes/literals



41

Incorrect/incomplete object value

Incorrect interlink

dbpedia:Oreye! !dbpedia-owl:postalCode! !“4360”!@en

Incorrect datatypes/literals


Results - Experts vs. Crowd

42


LD Expert MTurk Worker

58 80

3 weeks

4 days

1512

1073

0.38 0.73


LD experts MTurk Workers

Object values Fair!- required validation

Fair!- simple comparisons

Datatypes & literals Fair!- required validation

Poor!- inexperienced with

RDF

Interlinks Poor!- high effort required

Good!- high inter-rater

agreement

Summary — Experts vs. Crowd

43



Overview

44


review



Semi-!automated


quality


Research Questions

45



Research Questions

45

RQ2.6 How can we semi-automatically assess the quality of datasets and provide meaningful results to the user?!RQ3 How can we exploit Linked Data for building a use case and ensure good data quality?



Motivation — User Scenario

46



46

Healthcare!Policy maker



46



Which diseases?!Deaths per diseases?!

Where to allocate funds?

interested in


46





interested in

Databases!e.g. WHO, !

ClinicalTrials.gov

looks

at


46





interested in


ClinicalTrials.gov

looks

at

Data in disparate datasets, !in different formats!

Data quality problems!Subset of data!

Error-prone analysis etc.

analysis


46





interested in


ClinicalTrials.gov

looks

at

Data in disparate datasets, !in different formats!

Data quality problems!Subset of data!

Error-prone analysis etc.

analysis translates to Inadequate !allocations of!

funds

Use Case — Societal Progress Indicators

47

Evaluate the impact of Research & Development (R&D) — educational performance — on a country’s performance in:!

Economical!

Healthcare


Using Linked Data to evaluate the impact of Research and Development in Europe: a Structural Equation Model. Amrapali Zaveri, Joao Ricardo Nickenig Vissoci, Cinzia Daraio and Ricardo Pietrobon. ISWC 2013.


Datasets & Variables

48




48

4 datasets!

World Bank!

LinkedCT!

Scimago!

USPTO




48

4 datasets!

World Bank!

LinkedCT!

Scimago!

USPTO

17 variables !

Examples!

GDP (economical)!

Birth rate, death rate (healthcare)!

h-index (educational)



Methodology

49

World Bank Scimago

LinkedCT USPTO


*van Hage, W. R., Kauppinen, T., Graeler, B., Davis, C., Hoek- sema, J., Ruttenberg, A., and Bahls, D. (2014). SPARQL Package, v1.6. R Foundation for Statistical Computing.!* https://github.com/amrapalijz/R-LOD-SEM/blob/master/RSPARQL

extract


https://github.com/amrapalijz/R-LOD-SEM/blob/master/RSPARQL

Methodology

49

World Bank Scimago

LinkedCT USPTO

RSPARQL*



extract



Methodology

49

World Bank Scimago

LinkedCT USPTO

RSPARQL*



perform

extract



Methodology

49

World Bank Scimago

LinkedCT USPTO

Quality !Assessment

RSPARQL*



perform

extract



R2RLint tool*!

7 dimensions!

13 quality metrics !

Use case specific

Semi-automated Quality Assessment

50

*https://github.com/AKSW/R2RLint


https://github.com/AKSW/R2RLint

R2RLint tool*!

7 dimensions!

13 quality metrics !

Use case specific

Semi-automated Quality Assessment

50

Availability Completeness

Interlinking

Syntactic!validity!

Consistency

Interpretability

Representational conciseness

*https://github.com/AKSW/R2RLint


https://github.com/AKSW/R2RLint

Quality Assessment Results

51


Interlinking !completeness

Population !incompleteness

Inconsistency


Total no. detected

11/17 Variables

52

Latent variables

Observed variables

Educational!

performance

Number of articles (h) that have at least h citations (h-index)Total no. of documents published per country per yearHigh-technology export (HTE)

Healthcare!performance

Adolescent fertility rate (AFR)Birth rate (BR)Death rate (DR)Health expenditure public (HEP)Immunization DPT (IDPT)Immunization measles (IM)Mortality rate, infant (MR)

Economic performance

GDP per capita (current US$)


Methodology

53

World Bank

Scimago

Structural Equation Modeling

EFA*-CFA*-!EFA-CFA

Apply SEM to !hypothesis variables

Step I

Step II


apply


Methodology

53

World Bank

Scimago


EFA*-CFA*-!EFA-CFA

Apply SEM to !hypothesis variables

Step I

Step II

*EFA - Exploratory Factor Analysis!*CFA - Confirmatory Factor Analysis


apply


Theoretical Framework

54


Educational !performance


Economical!performance

correla

tion

correlation

correlation


55


https://github.com/amrapalijz/R-LOD-SEM/blob/master/sem_script.RLinked Data Quality Assessment and its Application to Societal Progress Measurement A. Zaveri

#Insert covariance matrix!var<-var(semdata)!cov<-cov(datanew)!cor<-cor(datanew)!#Acquire data!> data<-with(data,data.frame(hindex,noOfDocs,IDPT,IM,MR,! AFR,BR,DR,GDP,HEP,HET))!> ssemmodel<- specifyModel()!

#Latent Variables!> HealthCare->IDPT,efa14,NA; HealthCare->IM,efa11,NA; HealthCare-> MR,efa12,NA; HealthCare->AFR,efa13,NA;!….!#Running SEM model!> sem <- sem::sem(semmodel,cor, N=781)!> summary(sem,fit.indices=c("GFI", "AGFI", "RMSEA", "NFI","NNFI", "CFI", "RNI", "IFI", "SRMR", "AIC", "AICc"))!> modIndices(sem)!> qgraph(sem,cut = 0.8,gray=TRUE)

https://github.com/amrapalijz/R-LOD-SEM/blob/master/sem_script.R


56



Theoretical Framework

57


Educational !performance


Economical!performance

correla

tion

correlation

correlation

Conclusions

58



Conclusions

58

Performing robust statistical analysis on Linked Data can lead to important and meaningful insights on publicly available data for societal progress measurement.!

Importance of performing use-case driven data quality assessment of datasets before their utilisation.



Contributions

59

Contributions

59

Comprehensive survey !

18 data quality dimensions with definitions; 69 metrics!

12 tools compared according to 8 attributes!

Development and evaluation of data quality assessment methodologies!

User-driven - manual and semi-automated!

Crowdsourcing - experts vs. workers!

Semi-automated - application to a use case !

Consumption of Linked Data leveraging data quality

Future Work


Future Work

60

Standardized Quality assessment methodology for Linked Data


Future Work

60


Quality assessment tools for Linked Data


Future Work

60


Quality assessment tools for Linked Data

Detection as well as improvement of quality issues before utilization in Linked Data use cases


Conference Publications

61

Using Linked Data to evaluate the impact of Research and Development in Europe: a Structural Equation Model. Amrapali Zaveri, Joao Ricardo Nickenig Vissoci, Cinzia Daraio and Ricardo Pietrobon. ISWC 2013.!

Crowdsourcing Linked Data quality assessment. Maribel Acosta and Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Sören Auer and Jens Lehmann. ISWC 2013. !

User-driven Quality Evaluation of DBpedia. Amrapali Zaveri, Dimitris Kontokostas, Mohamed A. Sherif, Lorenz Bühmann, Mohamed Morsey, Sören Auer and Jens Lehmann. ISEMANTICS 2013.


Journal Publications

62

Quality assessment methodologies for Linked Data: A Survey. Amrapali Zaveri, Anisa Rula, Andrea Maurino, Ricardo Pietrobon, Jens Lehmann and Sören Auer. Semantic Web Journal 2015.!

Using Linked Data to build an Observatory of Societal Progress Indicators. Amrapali Zaveri, Joao Ricardo Nickenig Vissoci, Patrick Westphal, Jose Roberto Nascimento Junior, Luciano de Andrade, Cinzia Daraio, Jens Lehmann. Journal of Web Semantics 2014 (under review).!

Publishing and Interlinking the USPTO Patent Data. Amrapali Zaveri, Mofeed M. Hassan, Tariq Yousef, Sören Auer, Jens Lehmann. Semantic Web Journal 2014 (under review).


Publications

63

No. of publications: 34 (Google Scholar),16 (DBLP)!

Citations: 251 !

h-index: 9; i-10 index: 8 (Google Scholar)


Thank you for your attention !!

Questions?

[email protected]

64

mailto:[email protected]

Amrapali Zaveri Defense

Science

Transcript of Amrapali Zaveri Defense