Methods and techniques (2/6)
-
Upload
roberta-cuel -
Category
Documents
-
view
710 -
download
1
description
Transcript of Methods and techniques (2/6)
Methods and techniques to analyze and design incentivized
semantic applications
www.insemtives.eu
Roberta Cuel, University of Trento, IT and Markus Rohde, University of Siegen, DE
ISWC 2010
www.insemtives.eu04/14/2023 2
Why is my app not as successful as Facebook…
04/14/2023 www.insemtives.eu 2
• Motivation and incentives– Reciprocity – Reputation – Competition– Altruism– Self-esteem – Fun– Money
• This session is about the methods and techniques you can use – To study your scenario prior to
application design– To evaluate your incentives strategy
and adjust your application
www.insemtives.eu
Overview
• Basic notions and definitions• Methods and techniques for analysis, design
and evaluation• Case studies
04/14/2023 3
www.insemtives.eu
BASIC NOTIONS AND DEFINITIONS
Basic notions and definitionsMethods and techniques for analysis, design and evaluation
Case studies
04/14/2023 4
www.insemtives.eu
Starting with motivation…
• Basic tenets of organizational behavior – Performance : f(ability*motivation)
04/14/2023 5
www.insemtives.eu
Motivation and incentives
• Intrinsic vs extrinsic motivations• Incentives are ‘rewards’ assigned to performer
to make him/her commit– Can be totally uncorrelated to the nature of the
task – Need to be compliant with the values and beliefs
of the acting person
04/14/2023 604/14/2023 www.insemtives.eu 6
www.insemtives.eu
Example: DBPedia
The DBpedia community collaborates using the following tools– Mailing lists, Facebook group– Blog, quality assurance and bug tracking– Development
• “The framework is easily extended and we offer the possibility to do this yourself, increasing the likelihood of your desired data being included in the next DBpedia release.”
– DBpedia consulting (money)
04/14/2023 7
www.insemtives.eu04/14/2023 8
Typology of motivationsMotivations Internal
(embedded in structure, e.g., task, tools)
External(additional to structure, external re-inforcements)
Intrinsic(predispositioned in person, e.g., drives, needs, desires )
Fun, joy, gaming,interest, satisfaction, self-actualization, self-re-inforcement
Social appreciation, reputation, love, trust, social capital, community support
Extrinsic(additional to personal predispositions, extern re-inforcements )
Usability, sociability,Design-for-fun, curiosity,community-building support
Material/financial capital, money, rewards, prices, medals, credit points
Structure
Person
Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)
www.insemtives.eu
METHODS AND TECHNIQUES FOR ANALYSIS, DESIGN AND EVALUATION
Basic notions and definitionsMethods and techniques for analysis, design and evaluation
Case studies
04/14/2023 9
www.insemtives.eu
Overview of methods and techniques
• To analyze your scenario and design your application– Game theory, mechanism design– Participatory design and end-user development
• To evaluate an incentives strategy in its context– Interviews and questionnaires and field experiments – Heuristic evaluation, guideline reviews– Cognitive and pluralistic walk-throughs– Participatory/User evaluation
04/14/2023 1004/14/2023 www.insemtives.eu 10
www.insemtives.eu
Game theory
• Game theory is a formal way to analyze interaction among a number of rational agents who behave strategically– The rational agents: players involved in the situation (best choice)– A number of players: more than one– Rationality/payoffs: what are the players’ preferences over the
outcomes of the game– The interactions: one player’s behavior affects another – The rules: who moves when, what do they know, what can they do– The outcomes: what is the outcome of the game (for each move)
04/14/2023 11
www.insemtives.eu
Mechanism design
• Mechanism design is about how to translate game theory in effective behavior– To design rules such that a desired set of outcomes
happens– Alignment of interests between parties and production of
maximum social welfare
• Relevant variables– Structural and personal motivations– Goal and nature of good produced– Tasks and skills required– Social structure
04/14/2023 12
www.insemtives.eu
Analysis matrix
• A multidimensional tool that allows for an analysis of a particular task with respect to the variables relevant for mechanism design
04/14/2023 13
Goal Tasks Social structure Nature of good produced
Skill variety/level
Communication level
Variety of Hierarchy-neutral
Private good Skilled ability
Participation level
Specificity of Hierarchical Public good Competence
Clearness level Identification with
Common resourceClub good
www.insemtives.eu04/14/2023 14
The matrix in practice: semantic annotation
• Goal of the annotation or ontology population exercise• Task, or more typically, an ordered collection of tasks into
which the annotation exercise can be broken down• Social structure, a stylized and simplified set of social
relationships among the subjects participating in the exercise• Nature of good, a stylized description, in game-theoretical
terms, of the relationship between what good is produced and who consumes it
• Required skills of the agents to complete the annotation task
04/14/2023 www.insemtives.eu 14
www.insemtives.eu
The matrix in practice: crowd-sourcing of ontology evolution
• Goal: communication and participation level– Coordination (free-open)
• Interests are already aligned (free riding)• Private and public benefit coincide
• Task: variety vs specificity • Social structure: hierarchy neutral (large vs small
groups)• Nature of good: public good non-rival (use does not
limit use of others), non-exclusive (open access)• Skill variety/level: Skilled ability
04/14/2023 1504/14/2023 www.insemtives.eu 15
www.insemtives.eu
Participatory Design (PD)
• To develop software for the participation of end-users, we propose a participatory way of designing these software tools, integrating potential users by participatory design methods
• Participatory Design– Improvement of the participation of workers in software development
processes– The cooperation between software developers and end-users– Participate in IT development projects as experts of their own work
04/14/2023 16
www.insemtives.eu04/14/2023 17
Methods for PD
• Participatory-design projects combine– Design-by-doing methods – PD workshops– Scenarios – Different forms of prototyping (mockups, rapid
prototypes)– Work organization games – Ethnographic methods
www.insemtives.eu04/14/2023 18
End-User Development (EUD)
• EUD is about designing highly flexible systems that enable users to participate during the use of the system by adapting and modifying the tools according to their needs/preferences
• Goal: Empower end-users to develop and adapt systems themselves by designing these systems to be easy to– Understand– Learn– Use– Teach
www.insemtives.eu04/14/2023 19
How to design an incentivized applicationIdeally: field desk lab field
Analyze the domain and find yourselves in the matrixes Find the relevant point of that situation (goal and tasks)Focus on a small group of individuals (social structure)Analyze their motivation (internal/external intrinsic/extrinsic)Analyze the other relevant variables (nature of good being produced, kill variety/level)
Design a simplest possible model that can effectively support contributorsTest and get feedback Fine-tune the experiment and add other elements
www.insemtives.eu04/14/2023 20
Methods
• Domain observations (second-hand data)• Ethnography or qualitative face-to-face interviews • Questionnaires• Observations with selected individuals • Quantitative analysis (data collections)• Usability-design methods
04/14/2023 www.insemtives.eu 20
www.insemtives.eu
CASE STUDIES
Basic notions and definitionsMethods and techniques for analysis, design and evaluation
Case studies
04/14/2023 21
www.insemtives.eu
Examples from our project
• Telefónica Investigación y Desarrollo (TID - Spain): Knowledge Intranet Platform
• Pepper’s Ghost (PGP-UK): Virtual Games/Worlds• Seekda! (Austria): Web Services Search Portal
04/14/2023 22
www.insemtives.eu04/14/2023 23
Field work: TID
• Domain analysis– Site visit, semi-structured, qualitative interviews
• Communication processes• Existing usage practices and problems• Existing tools/solutions• Semantic annotation solutions
– Tape recording, transcription– Data analysis per ex-post categorization
• Focus group discussion– Usability lab tests– Expert walkthroughs
www.insemtives.eu04/14/2023 24
Field work: TID (2)
04/14/2023 www.insemtives.eu 24
• Find TID in the matrix– Goal: communication and participation (contribute to the
knowledge portal) – Social structure (various structures co-exist)
• Strongly hierarchical organization (control is an issue) • working groups and community of experts
– Nature of good: public good vs. private, club goods– Skill variety/level: Skilled ability (knowledge workers)– Motivations: fun, visibility, reputation, promotion, money
www.insemtives.eu04/14/2023 25
Field work: PGP• Domain analysis– Data analysis (Benchmark: Galaxyzoo, Moonzoo)– Usability lab tests and expert walkthroughs– Collect data on users such as logs files
• Find PGP in the matrix– Goal: participation level– Task: specific– Social structure: hierarchy neutral– Nature of good: public good non-rival (use does not limit
use of others), non-exclusive (open access)– Skill variety/level: non skilled ability
www.insemtives.eu04/14/2023 26
Field work: SEEKDA
• Domain analysis– Interviews and workshops – Replicating Telefonica model
• Find SEEKDA in the matrix:– Goal: communication and participation level– Task: variety vs. specificity – Social structure: hierarchy neutral (mainly)– Nature of good: public good non-rival (use does not limit
use of others), exclusive, or club good– Skill variety/level: Skilled ability