Crowdsourcing ontology engineering Elena Simperl Web and Internet Science, University of Southampton...
-
Upload
macey-higley -
Category
Documents
-
view
215 -
download
0
Transcript of Crowdsourcing ontology engineering Elena Simperl Web and Internet Science, University of Southampton...
Crowdsourcing ontology engineeringElena SimperlWeb and Internet Science, University of Southampton 11 April 2013
2
Overview• "online, distributed problem-
solving and production model“ [Brabham, 2008]
• Varieties: wisdom of the crowds/collective intelligence, open innovation, human computation...
• Why is it a good idea?
– Cost and efficiency savings – Wider acceptance, closer to
user needs, diversity
• Approaches
– Collaborative ontology engineering
– Challenges/competitions
– Games with a purpose – Microtask/paid
crowdsourcing
• In combination with automatic techniques
3
Crowdsourcing ontology alignment• Experiments using MTurk, CrowdFlower and established benchmarks
• Enhancing the results of automatic techniques
• Fast, accurate, cost-effective
[Sarasua, Simperl, Noy, ISWC2012]
CartP301-304
100R50PEdas-Iasted
100R50PEkaw-Iasted
100R50PCmt-Ekaw
100R50PConfOf-Ekaw
Imp301-304
PRECISION 0.53 0.8 1.0 1.0 0.93 0.73
RECALL 1.0 0.42 0.7 0.75 0.65 1.0
Open questions• Quality assurance and evaluation
• Incentives and motivators
• Choice of crowdsourcing approach and combinations of different approaches
• Reusable collection of algorithms for quality assurance, task assignment, workflow management, results consolidation etc
• Schemas recording provenance of crowdsourced data
• Descriptive framework for classification of human computation systems
– Types of tasks and their mode of execution– Participants and their roles – Interaction with system and among participants– Validation of results– Consolidation and aggregation of inputs into complete solution