Cognition for the Semantic Web

Involving Humans in Semantic Data Management

Irene Celino (irene.celino@cefriel.com)

CEFRIEL – Milano, Italy

Agenda Methods to involve people (a.k.a. crowdsourcing and its brothers)

Motivation and incentives (a.k.a. let’s have fun with games)

Crowdsourcing and the Semantic Web (a.k.a. this is SSSW after all…)

Methods to involve people

What goals can humans help machines to achieve? Which scientific communities “exploit” people? How to involve a crowd of persons?

- Citizen Science

- Crowdsourcing

- Human Computation

Wisdom of crowds [1]

“Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations”

Criteria for a wise crowd

Diversity of opinion (importance of interpretation)

Independence (not a “single mind”)

Decentralization (importance of local knowledge)

Aggregation (aim to get a collective decision)

The are also failures/risks in crowd decisions:

Homogeneity, centralization, division, imitation, emotionality

Citizen Science [2]

Problem: a scientific experiment requires the execution of a lot of

simple tasks, but researchers are busy Solution: engage the general audience in solving those tasks, explaining that

they are contributing to science, research and the public good

Example: https://www.zooniverse.org/

Crowdsourcing [3]

Problem: a company needs to execute a lot of simple tasks, but cannot afford

hiring a person to do that job

Solution: pack tasks in bunches (human intellingence tasks or HITs)

and outsource them to a very cheap workforce through an online platform

Example: https://www.mturk.com/

Human Computation [4]

Problem: an Artificial Intelligence algorithm is unable to achieve an

adequate result with a satisfactory level of confidence

Solution: ask people to intervene when the AI system fails, “masking” the task

within another human process

Example: https://www.google.com/recaptcha/

Spot the difference…Similarities:

- Involvement of people

- No automatic replacement

Variations:

- Motivation

- Reward (glory, money, passion)

Hybrids or parallel!Citizen Science

CrowdsourcingHuman

Computation

Motivation and incentives

Apart from extrinsic rewards (money, prizes, etc.) what are the intrinsic incentives we can adopt to motivate people? How can we leverage “fun” through games and game-like applications?

- Gamification

- Games with a Purpose (GWAP)

Gamification [5,6]

Problem: motivate people to execute boring or mandatory tasks (especially

in cooperative environments) that they are usually not very happy to do

Solution: introduce typical game elements (e.g. points, badges, leaderboards) within the more

traditional processes and systems

Example: https://badgeville.com/

Games with a Purpose (GWAP) [7,8]

Problem: same of Human Computation (ask

humans when AI fails)

Solution: hide the task within a game, so that users are motivated by game

challenges, often remaining unaware of the hidden purpose, task solution

comes from agreement between players

Example: http://www.gwap.com/

Crowdsourcing and the Semantic Web

Can we involve people in Semantic Web systems? What semantic data management tasks can we effectively “outsource” to humans?

Why Crowdsourcing in the Semantic Web?

Knowledge-intensive and/or context-specific character of Semantic Web tasks:e.g., conceptual modelling, multi-language resource labelling, content annotation with ontologies, concept/entity similarity recognition, …

Crowdsourcing can help to engage users and involve them in executing tasks:e.g., wikis for semantic content authoring, folksonomies to bootstrap formal ontologies, human computation approaches, …

Semantic Crowdsourcing [9]

Many tasks in Semantic Web data management/curation can exploit Crowdsourcing

Fact level

Schema level

Collection Creation CorrectionValidation Filtering Ranking Linking

Conceptual modelling

Ontology population

Quality assessment

Ontology re-engineering

Ontology pruning

Ontology elicitation

Knowledge acquisition

Ontology repair

Knowledge base update

Data search/ selection Link

generation

Ontology alignment

Ontology matching

Focus on Data LinkingCreation of links in the form of RDF triples (subject, predicate, object)

Within the same dataset (i.e. generating new connections between resources of the same dataset)

Across different datasets (i.e. creating RDF links, as named in the Linked Data world)

Notes: Generated links can have an associated score σ ∈ 0,1 expressing a sort of “confidence” in the truth value or in the “relevance” of the triple

In literature, data linking often means finding equivalent resources (similarly to record linkage in database research), i.e. triples with correspondence/match predicate (e.g. owl:sameAs) in the following, data linking is intended in its broader meaning (i.e. links with any predicate)

Subject: 𝑟𝑠 ∈ {Milano cultural heritage assets}Predicate: foaf:depiction Goal: create <asset> foaf:depiction <photo>

Object: ? ∈ {photos}

Knowledge acquisition

Citizen science campaign

http://bit.ly/picmiapp

Level badges as intrinsic incentives

(gamification)

Contributions with attribution/recognition

Set of all links: <asset> foaf:depiction <photo>

Goal: assign score 𝜎 to rank links on their “recognisability”/“representativeness”

Link ranking [10]http://bit.ly/indomilando

Pure GWAP with hidden purpose

Points, badges, leaderboard as intrinsic reward

The score 𝜎 is a function of 𝑋 𝑁where 𝑋 is the no. of successes (=recogni-tions) and 𝑁 the no. of trials of the Bernoulli process (guess or not guess) realized by the game

Link ranking is a result of the “agreement”

between players

Link validation [11]

http://bit.ly/foss4game

Set of links: <land-area> clc:hasLandCover <land-cover>

Automatic classifications: <land-cover-assigned-by-DUSAF> ≠ <land-cover-assigned-by-GL30>

Goal: assign score 𝜎 to each link to discover the “right” land cover class

Pure GWAP with not-so-hidden purpose (played by “experts”)

Points, badges, leaderboard as intrinsic reward

https://youtu.be/Q0ru1hhDM9Q

A player scores if he/she guess one of the two

disagreeing classifications

Score 𝜎 of each link is updated on the basis of players’ choices (incremented if link selected, decremented if link not selected)

When the score of a link overcomes a threshold 𝜎 ≥ 𝑡 , the link is considered “true”

Link validation is a result of the “agreement”

between players

Caveat 1: Mice and Men (or: keep it simple)

Crowdsourcing workers behave like mice [12]

Mice prefer to use their motor skills (biologically cheap, e.g. pressing a lever to get food) rather than their cognitive skills (biologically expensive, e.g. going through a labyrinth to get food)

Workers prefer/are better at simple tasks (e.g. those that can be solved at first sight) and discard/are worse at more complex tasks (e.g. those that require logics)

Crowdsourcing tasks should be carefully designed

Tasks as simple as possible for the workers to solve

Complex tasks together with other incentives (e.g. variety/novelty)

Suggestion 1: Divide et impera (or: Find-Fix-Verify)

Find-Fix-Verify crowd programming pattern [13]

A long and “expensive” task…Summarize a text to shorten its total length

…is decomposed in more atomic tasks…1. find sentences that need to be shortened

2. fix a sentence by shortening it

3. verify which summarized sentence maintains original meaning

…and the complex task is turned into a workflow of simple tasks, and each step is outsourced to a crowd

Caveat 2: aggregation and disagreementAre all contributors “created equal”?

Contributions/results on the same task are usually aggregated on different workers (“wisdom of crowds”, “collective intelligence”)

Update formula for 𝜎 score should weight contributions differently by including some evaluation of contributors’ reliability (e.g. gold standard)

Is there always a “right answer”? Or is there a “crowd truth”? [14]

Not always true/false, because of human subjectivity, ambiguity and uncertainty

Disagreement across contributors is not necessarily bad, but a sign of: different opinions, interpretations, contexts, perspectives, …

Suggestion 2: compare and contrastFrom “wisdom of crowds” to “wisdom of the crowdsourcing methods”: different approaches to solve the same problem could be put in parallel to compare results

Which is the best crowdsourcing approachfor a specific use case?

Which is the most suitable crowd?

Is crowdsourcing better/faster/cheaper than automatic means (e.g. AI)?

Examples:

Detecting quality issues in DBpedia [15]: find-verify strategy with different crowds (experts and workers) behind the find step

Employing people for ontology alignment [16]: outcomes comparable to results of OAEI systems

input task

output solution

Citizen Science

Human Computation

Crowdsourcing

Automatic/machine computation

Bibliography (1/2)

SSWS 2016 - Bertinoro, 22th July 2016

[1] James Surowiecki. The wisdom of crowds, Anchor, 2005.

[2] Alan Irwin. Citizen science: A study of people, expertise and sustainable development. Psychology Press, 1995.

[3] Jeff Howe. Crowdsourcing: How the power of the crowd is driving the future of business. Random House, 2008.

[4] Edith Law and Luis von Ahn. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(3):1–121, 2011.

[5] Jane Mc Gonigal. Reality is broken: Why games make us better and how they can change the world. Penguin, 2011.

[6] Kevin Werbach and Dan Hunter. For The Win: How Game Thinking Can Revolutionize Your Business.Wharton Digital Press, 2012.

[7] Luis Von Ahn. Games with a purpose. Computer, 39(6):92–94, 2006.

[8] Luis Von Ahn and Laura Dabbish. Designing games with a purpose. Communications of the ACM, 51(8):58–67, 2008.

Bibliography (2/2)

SSWS 2016 - Bertinoro, 22th July 2016

[9] Cristina Sarasua, Elena Simperl, Natasha Noy, Abraham Bernstein, Jan Marco Leimeister. Crowdsourcing and the Semantic Web: A Research Manifesto, Human Computation 2 (1), 3-17, 2015.

[10] Irene Celino, Andrea Fiano, and Riccardo Fino. Analysis of a Cultural Heritage Game with a Purpose with an Educational Incentive, ICWE 2016 Proceedings, pp. 422-430, 2016.

[11] Maria Antonia Brovelli, Irene Celino, Monia Molinari, Vijay Charan Venkatachalam. A crowdsourcing-based game for land cover validation, ESA Living Planet Symposium 2016 Proceedings, 2016.

[12] Panos Ipeirotis. On Mice and Men: The Role of Biology in Crowdsourcing, Keynote talk at Collective Intelligence, 2012.

[13] M. Bernstein, G. Little, R. Miller, B. Hartmann, M. Ackerman, D. Karger, D. Crowell, K. Panovich. Soylent: A Word Processor with a Crowd Inside, UIST Proceedings, 2010.

[14] Lora Aroyo, Chris Welty. Truth is a Lie: 7 Myths about Human Annotation, AI Magazine 2014.

[15] Maribel Acosta, Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Fabian Flöck, Jens Lehmann. Detecting Linked Data Quality Issues via Crowdsourcing: A DBpedia Study, Semantic Web Journal, 2016.

[16] Cristina Sarasua, Elena Simperl, Natasha Noy. Crowdmap: Crowdsourcing ontology alignment with microtasks, ISWC 2012 Proceedings, pp. 525-541, 2012.

Other relevant work (1/2)

SSWS 2016 - Bertinoro, 22th July 2016

Cultural heritage annotation: C. Wieser, F. Bry, A. Bérard and R. Lagrange. ARTigo: building an artwork search engine with games and higher-order latent semantic analysis. In First AAAI Conference on Human Computation and Crowdsourcing, 2013.

Ranking triples for relevance:J. Hees, T. Roth-Berghofer, R. Biedert, B. Adrian and A. Dengel. BetterRelations: using a game to rate linked data triples. In Annual Conference on Artificial Intelligence, 2011.I. Celino, E. Della Valle and R. Gualandris. On the effectiveness of a Mobile Puzzle Game UI to Crowdsource Linked Data Management tasks. In CrowdUI workshop, 2014.

DBpedia data cleansing:J. Waitelonis, N. Ludwig, M. Knuth and H. Sack. WhoKnows? Evaluating Linked Data Heuristics with a Quiz that cleans up DBpedia. Interactive Technology and Smart Education, 8(4), 2011.

Urban Games on spatial knowledge:I. Celino, S. Contessa, M. Corubolo, D. Dell'Aglio, E. Della Valle, S. Fumeo and T. Krüger. Linking Smart Cities Datasets with Human Computation – the case of UrbanMatch, In ISWC 2012.I. Celino, D. Cerizza, S. Contessa, M. Corubolo, D. Dell'Aglio, E. Della Valle and S. Fumeo. Urbanopoly – a Social and Location-based Game with a Purpose to Crowdsource your Urban Data, Proceedings of SoHuman 2012, 2012.

Other relevant work (2/2)

SSWS 2016 - Bertinoro, 22th July 2016

Crowdsourcing ontology engineering tasks or SPARQL querying:N. Noy, J. Mortensen, M. Musen and P. Alexander. Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow. In WebSci 2013.G. Wohlgenannt, M. Sabou and F. Hanika. Crowd-based ontology engineering with the uComp Protégé plugin. Semantic Web, 7(4), 2016.M. Acosta, E. Simperl, F. Flöck and M.E. Vidal. HARE: A Hybrid SPARQL Engine to Enhance Query Answers via Crowdsourcing. K-CAP 2015.

Crowdsourced contributions and crowd quality:R. Kern, H. Thies, C. Zirpins and G. Satzger,. Dynamic and Goal-Based Quality Management for Human-Based Electronic Services. Int. J. Cooperative Inf. Syst. 21, 1, 2012.T. Schulze, S. Krug and M. Schader. Workers’ Task Choice in Crowdsourcing and Human Computation Markets. In ICIS Association for Information Systems, 2012.D. Difallah, G. Demartini and P. Cudré-Mauroux. Pick-a-crowd: tell me what you like, and I'll tell you what to do. In WWW 2013.M. Feldman and A. Bernstein. Cognition-based Task Routing: Towards Highly-Effective Task-Assignments in Crowdsourcing Settings. In ICIS 2014.

Mixed crowdsourcing approaches:M. S. Bernstein. Crowd-Powered Systems. KI 27, 1, 69–73, 2013.