The Promise of a Better Connected Digital World:

Philippe Cudré-MaurouxeXascale Infolab, University of Fribourg

Switzerland

Verisign Labs Distinguished Speakers SeriesVerisign Labs, Reston–USA

December 13, 2013

Christophe GuéretVU University / DANS

The Netherlands

Data Registry Systems Without the Web

Entities2

Entity Data

• Semi-structured, interlinked descriptions of shared instances– Persons– Objects– Software– Locations– Sensors– …

3

Entities as Mediation

• Rising paradigm– Store information at the entity granularity– Integrate information by inter-linking entities

• Advantages?– Coarser granularity compared to keywords

• More natural, e.g., brain functions similarly (or is it the other way around?)

– Denormalized information compared to RDBMSs• Schema-later, heterogeneity, sparsity• Pre-computed joins, “Semantic” linking

• Drawbacks?

4

Prominence of Entity-Powered Apps

– Collaborative Editing (Wikipedia’s wikidata)– Social Networks (Facebook’s Open Graph)– Serious Networks (LinkedIn’s Business Graph)– Web Search (Google’s Knowledge Graph)– Software Integration (Yahoo!’s WOO)– Question Answering (IBM’s Watson)– Dynamic Websites (BBC’s London Olympics)– Open Data (data.gov.uk, linkeddata.org)– Most of our own applications (exascale.info)– etc. etc.

5

Problem: Limited Access to Entities (1)

• 70+% of the world’s population has no or very limited access to the Web

[Ahmed Shams 2013]6

Problem: Limited Access to Entities (2)

• Even in developed countries, deploying collaborative entity-editing platforms is technically exceedingly challenging– Local/Global QoS to serve arbitrary entity data

• Performance, scale-out

– Collaborative aspects• Transactions, versioning, integration

– Offline / mobile concerns• Caching / replication / serializability

7

Potential Building Blocks?

• … for a hybrid online/offline, collaborative entity registry:– DNS3 (never meant for entity data)– DOA (awkward Web integration, limited features)– RDBMSs (ACID? Impedance mismatch, limited perf.)– P2P / decentralized CDNs (performance issues)– Native RDF Stores (too expressive; scalability / perf. issues)– (Structured) Inverted Indices (no transactions; slow updates)– noSQL key-value / document stores

(wrong PACELC trade-offs; (some) performance issues)

8

[Iliya Enchev 2012ISWC 2013]

Our Solution: ERS, theEntity Registry System

• Three-tier solution to deploy entity-powered apps– Flexible

• Seamlessly reconcile entities in local / ad-hoc / global modes

– Collaborative• Transactional consistency, data versioning

– Scalable• Bridges, scale-out servers, tunable consistency

– Open-source• https://github.com/ers-devs

9

ERS Architecture (1)

• Contributors: Contributors read and edit the contents of the registry. They may create and delete entities, look for entities, and contribute to the entities’ descriptions.

• Bridges: Bridges do not directly contribute to the contents of the registry. They are used to connect isolated closed networks and improve the availability of the descriptions shared by the contributors.

• Aggregators: Some use-cases may require the presence of global servers that contains a copy of all the data provided individually by the contributors. The global server provides a single entry point to the registry.

10

ERS Architecture (2)

11

www www

Sample Deployment [Videos]

12

ERS Data & API

• Data: flexible RDF quads serialized as – JSON documents (contributors, bridges)– Key, value pairs (aggregators)

• Atomic & serializable operations through various locking granularities– Insert entity (IE), Insert property (IP), Update property (UP), Delete

property (DP), Delete entity (DE), Shallow entity copy (SC), Deep entity copy (DC), Insert link between two entities (IL), Delete link between two entities (DL)

=> Consistency

13

Unique Technical Features

I. Seamless, best-effort entity synchronization– Local, ad-hoc, global modes

II. Fault-tolerance and decentralization– Property replication, no single point of failure

III. Built-in versioning and provenance– Collaborative entity editing made easy

IV. Linear scalability– Tunable consistency levels

14

Performance: Distributed Locking (1)

• Decentralized, multi-granular locking protocol for transactional consistency on top of persistency layers

15

Performance: Distributed Locking (2)

• Fault-tolerant, though Paxos-like algorithms limit horizontal scalability

16

Performance: Optimistic Concurrency (1)

• ERS typically operates on insert-heavy low-conflict workloads– Most of the time new entities are inserted and properties added

• Goal: separate validation from write operations– Per worker TX management– Distributed ID generator for consistent commits

17

Tunable Consistency in ERS• Weak Writes: For each TX a CID is acquired and a new record

is written, write is not validated

• Strong Writes: For each TX a CID is acquired and a new record is written. After the write, a read verifies the visibility; if the record is not visible the write is performed again

• Write validation: Forward chaining of records based on the highest CID, last writer wins

111

Inserting 111 is possible using weak writes, but the write cannot be validated

18

OC – Execution Stack

• Breakdown of a single write operation with tunable consistency

20

Performance: Optimistic Concurrency (2)

=> Linear scalability even for write-heavy workloads

Ongoing Deployments (1)

• Swiss-Dutch local/global social messaging

21

Cloud hosted Aggregator

Bridge inAmsterdam (VUA)

Bridge inSwitzerland (Exascale)

Contributors on VUA internal network

Contributors on Exascale Infolabinternal network

Ongoing Deployments (2)

• Test deployments on new affordable devices

Wandboard(ultra low power computer)

SmilePlug(cloud-based learning)

Earl (backcountry survival tablet)

Ongoing Deployments (3)

• Entity-powered apps for the Sugar Learning Platform

23

Ongoing Deployments (4)

• ERS for Ambient Assisted Living of elderly persons in tropical environments[AAL research group @ VU]

24

Conclusions• The Web is becoming entity-centric

–Land of opportunities for new registries–Urgent needs for developing countries

• ERS is a unique, open-source entity registry solution supporting

–Local / ad-hoc / global modes–Collaborative editing and entity versioning–Tunable consistency levels–Linear scalability

• Series of ongoing deployments–Stay tuned for more results and lessons learnt

Big Thanks to the whole ERS Team

… and to our MSc students:–Iliya Enchev and Ahmed Shams

Dr. Christophe Guéret

Dr. Marat Charlaganov

C. Dinu & Pepijn Kroes

Prof. Dr. Philippe Cudré-Mauroux

Dr. Martin Grund

Teodor Macicas

Dutch Team @ DANS Swiss team @ XI

And Special Thanks to…

• Scott Hollenbeck, Debra Anderson, Allison Mankin & the Internet Infrastructures Grant team

• Dr. Burt Kaliski and his team

• Vincenzo Russo, Benoit Perroud, Romain Cholat and the whole Verisign Fribourg office

… for their continued support

References• P. Cudré-Mauroux, G. Demartini, D.E. Difallah, A.E. Mostafa, V. Russo, and M. Thomas. A

Demonstration of DNS3: a Semantic-Aware DNS Service. ISWC 2011.

• P. Cudré-Mauroux, G. Demartini, I. Enchev, C. Gueret and B. Perroud: Downscaling Entity Registries for Ad-Hoc Environments. Downscale 2012.

• M. Charlaganov, P. Cudré-Mauroux, C. Dinu, C. Guéret, M. Grund, T. Macicas: Demonstrating The Entity Registry System: Implementing 5-Star Linked Data Without the Web. ISWC 2013.

• P. Cudré-Mauroux, I. Enchev, S. Fundatureanu, P.T. Groth, A. Haque, A. Harth, F. Keppmann, D.P. Miranker, J. Sequeda, M. Wylot: NoSQL Databases for RDF: An Empirical Evaluation. ISWC 2013.

• M. Charlaganov, P. Cudré-Mauroux, C. Dinu, C. Guéret, M. Grund, T. Macicas: The Entity Registry System: Implementing 5-Star Linked Data Without the Web. CoRR abs 2013.

• M. Charlaganov, P. Cudré-Mauroux, C. Dinu, C. Guéret, M. Grund, P. Kroes, and T. Macicas: Collaboratively Editing an Entity Registry in Poorly Connected Environments.

CAiSE 2014 [submitted].

28

Further Entity Research @ XI• R. Prokofyev, G. Demartini and P. Cudré-Mauroux: Effective

Named Entity Recognition for Idiosyncratic Web Collections. WWW 2014.

• G. Demartini, D.E. Difallah., and P. Cudré-Mauroux: Large-scale linked data integration using probabilistic reasoning and crowdsourcing. The VLDB Journal, 2013.

• A. Tonon, M. Catasta, G. Demartini, P. Cudré-Mauroux, and K. Aberer: TRank: Ranking Entity Types Using the Web of Data. ISWC 2013.

• A. Tonon, G. Demartini, and P. Cudré-Mauroux: Combining inverted indices and structured search for ad-hoc object retrieval. SIGIR 2012.

• G. Demartini, D.E. Difallah, and P. Cudré-Mauroux: ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. WWW 2012.

29

Thanks a lot for your attention

30

http://exascale.info