Insemtives at the KIT

Web 2.0 and incentives for human-driven contributions

Roberta Cuel Univeristy of Trento and KIT

[email protected]@kit.edu

2/23/11 2

Motivations in the Web 2.0

2/23/11 2

• Motivation and incentives– Reciprocity – Reputation – Competition– Altruism– Self-esteem – Fun– Money

Motivation

• Basic tenets of organizational behavior:

• Performance : f (ability*motivation)

• Incentives Motivation Performance

www.insemtives.eu

Performance components:

Task (abilities needed to get the job done)

Contextual (abilities to get the job done here and now organizational citizenship behavior or prosocial organizational behaviors (altruism, politeness, etc.))

Ethical (ability to do the “right” thing)

Technology (skill vs. standardization)

What is motivation?

• Etymology: motivation latin movere “move”

• Psychological meaning: internal mental state pertaining to initiation, direction, persistence, intensity and termination of behavior

• Managerial meaning: activity implemented to induce others to produce results

• So we will deal both with reasons for behavior and processes that cause those behaviors• Content theories of motivation• Process theories of motivation

Content theories of motivation• Need theories

• People will act to satisfy a gap between the present condition and a desired state• Maslow identifies 5 kinds of need: Physiological, Safety, Belonging, Esteem, Self-

actualization• ERG identifies 3 kinds of need: Existence (physiological+security), Relatedness (that

regard relationships and belonging), Growth (Esteem and self-actualization)

• Herzberg’s “two factor” theory• Hygiene factors: create dissatisfaction if they are not present (e.g. salary, job security,

relationship with supervisor), do not motivate per se• Motivators: create satisfaction when they are present and thus willingness to work

harder (e.g. responsability, achievement, work itself, growth possibilities)

• McClelland’s achievement-power-affiliation theory• “Achievement- oriented people”: prefer own effort, direct feedback, medium risk

situations• “Power-oriented people”: prefer control over others’ effort, elicit strong emotions,

concern for reputation• Low- high affiliation needs make people more competition vs. cooperation oriented.

• Job characteristic approach

Job characteristics approach

Principle: nature of the work affects motivation and performance

Core job dimensions

Critical psychological states

Personal and work outcomes

Feedback

Autonomy

Skill variety

Task identity

Task significance

Meaningfulness of work

Responsibility for work outcomes

Knowledge on results

Higher internal work motivation

High quality work performance

High satisfaction

Low absenteeism

Process theories on motivation

• Reinforcement theory• Positive and negative reinforcement• Punishment or extinction (no rewards): Communication processes reduces,

Interpersonal tension increases, Exit strategies

• Goal-setting theory: Specific goals > better results

• Group discussion clarifying the goal

• Expectancy theory (Effort-performance or Performance-outcome expectancy)

• Organizational justice theory (Distributive justice (fair treatment with respect to effort-reward balance) or procedural justice (fair treatment in terms of how decisions are made about things that affect them in the workplace))

Low

Per

form

ance

hi

gh

Low difficulty of a goal imp.ble

Frustration

Porter and Lawler model

Value of rewards

Perceived effort -reward probability

Effort Performance

Abilities

Organizational context

Extrinsic rewards

Intrinsic rewards

Satisfaction

Equity perception

Types of motivations

Motivations Internal

(embedded in structure, e.g., task, tools)

External

(additional to structure, external re-inforcements)

Intrinsic

(predispositioned in person, e.g., drives, needs, desires )

Fun, joy, gaming,interest, satisfaction, self-actualization, self-re-inforcement

Social appreciation, reputation, love, trust, social capital, community support

Extrinsic

(additional to personal predispositions, extern re-inforcements )

Usability, sociability,Design-for-fun, curiosity,community-building support

Material/financial capital, money, rewards, prices, medals, credit points

Structure

Person

Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)

Intrinsic / Extrinsic motivations

Kaufman, Schulze, Veit (Mannheim University)

Game theory

Game theory is a formal way to analyze interaction among a number of rational agents who behave strategically

– The rational agents: players involved in the situation (best choice)– A number of players: more than one– Rationality/payoffs: what are the players’ preferences over the

outcomes of the game– The interactions: one player’s behavior affects another – The rules: who moves when, what do they know, what can they do– The outcomes: what is the outcome of the game (for each move)

Mechanism design

• Mechanism design is about how to translate game theory in effective behavior– To design rules such that a desired set of outcomes happens– Alignment of interests between parties and production of maximum social

welfare

• A selection of relevant variables– The goal: the goal of the software and its relation with goals and interests

of its users – The tasks: how much effort the task requires from the user and which

competences and abilities are required to carry on the task – The social structure: a stylized and simplified set of social relationships

among the users. – The nature of good: who can benefit from individual contributions

The incentive context matrix

Motivations: another study

In the case of MTurk

Design systems fostering users to provide good quality content

• Ideally: field desk lab field• Analyze the domain and find yourselves in the

matrixes • Find the relevant point of that situation (goal and tasks)

• Focus on a small group of individuals (social structure)

• Analyze their motivation (internal/exterrnal intrinsic/extrinsic)

• Analyze the other relevant variables (nature of good being produced, kill variety/level)

• Design a simplest possible model that can effectively support contributors

• Test and get feedback • Fine-tune the experiment and add other elements

A procedural ordering of methods to develop incentive compatible applications

Workshop and interviews reports

• Domain observations (second-hand data)• Ethnography or qualitative face-to-face

interviews • Questionnaires• Observations with selected individuals • Quantitative analysis (data collections)

Field experiment

As lab experiment, but …•… real users•… real context•… real tasks•… incentives have work related consequences •… evaluation is conducted on these

consequences

A CASE STUDY

Telefónica Investigación y Desarrollo (TID - Spain)

Field and domain analysis

• Domain analysis– Site visit, semi-structured, qualitative interviews

• Communication processes• Existing usage practices and problems• Existing tools/solutions• Semantic annotation solutions

– Tape recording, transcription– Data analysis per ex-post categorization

• Focus group discussion– Usability lab tests– Expert walkthroughs

Workshop and interviews report

• Various styles– In house and remote workers – Composition of groups

• In some groups people are working together for long time • Some other projects started only few months ago

• People are sharing docs only inside their group – Directory is more than enough– No one is looking for information in the portal – They use networks to

• Get information • Solve problems

• Strongly hierarchical organization – Personal benefits– Control is an issue– Reasons for annotation: Are mandatory (quality); Altruism; Fun;

– Social structure (various structures co-exist)• Strongly hierarchical organization (control is an issue) • Working groups and community of experts

– Nature of good: public good vs. private, club goods– Skill variety/level: Skilled ability (knowledge workers)– Motivations: fun, visibility, reputation, promotion, money

The TID matrix & motivations

Why mechanism design?

• The case of TID:– Annotation is a public good– Free riding problem– The problem of efficient allocation of time between

main task and annotating activity

• We need to design the “game” in a way that permits to achieve the outcome in annotations but does not distruct too much employees from their main job

The prototype creation

PD workshops and HCI analysis

Lab experiment

•Test two rewarding/incentives systems•Pay per click:

– 0,03 € per tag added (up to 3 € maximum).

•Winner takes all model: – The person who adds the higher number of

tags/annotation wins 20€

(Participation fee – 5 €)

The experiment (setting)

36 students (Random assignment to the two “treatments”)

Individual task: annotation of images

•Clear set of Instructions

•Training (guided) session to give basic understanding of

annotation tool

•8 minutes clocked session (time pressure)

Goal: produce maximum amount of tags in allotted time

on a random set of images

The lab

www.insemtives.eu 36

The experiment: screenshots

2/23/11 www.insemtives.eu 38

First results


First results In WTA treatment, 76 % of subjects make more annotations than the average number of annotations in PPT scenario.

Anyway, there are several subjects in both scenarios that make few annotations.

This suggests that WTA incentive system doesnot provide the same incentives to all subjects.

The results

2/23/11www.insemtives.eu

44

This suggests that WTA scenario spurs subjects to perform at their very best. At the same time, under PPT incentive system subjects seem to be less interested in the final result and tend to coordinate at lower levelsof productivity.

3

5

Quality of data

The number of images annotated by users in both treatments tend to be the same, subjects annotate 11 images on average.

•PPT • 134 tags repeated only 2 times • 437 unique tags

•WTA• 118 tags repeated only 2 times • 390 unique tags

The costs and scalability

• The real scenario with 19/17 subjects– PPT scenario = 27.03 €– WTA scenario = 20 €

• What if 40 subjects or more ?– PPT scenario = 56 €; 0.03 € per annotation – WTA scenario = 20 €; 0.008 € per annotation– Note that the prize in the WTA scenario should be

adjusted according to the chances of winning it.

Some biases

• Students are – Volunteers who are used participating in experiments– Strong web users and game players – Paid to show up

• Quality of the tags– Quality of tagging has been controlled for: no

obvious ‘mistakes’ or ‘cheating’

Prototype refinement

“Incentivizing the tool”

Next steps: Telefonica I+D

• Replicate experiment with real users– Change 1: task becomes relevant in terms of practical

usefulness for users (search):• Effort directed to producing a good (tags) that are not

consumed by users (used to achieve other goals) à change structure of the game to let users exploit tagging to achieve results (treasure hunt!)

– Change 2: task has social implications• Providing information about contribution by others• Providing information about relative standing of the user

compared to the community

Next steps: Telefonica I+D

• Replicate experiment with real users– Change 3: Mimic the social structure in the company:

• Run experiment with teammates• Use real tasks• Try alternative pay for performance schemes (users produce

tags not to get money but to use tags to perform other tasks)

Field Desk Lab Field

Example: MovieLens

• http://www.movielens.org is an online movie recommender system that invites users to rate movies and in return makes personalized recommendations and predictions for movies the user has not already rated.

• Personalized recommendations are based on collaborative filtering technology

Background knowledge: social information might affect behaviors

• People lean toward social comparisons in situations that are ambiguous

• When information regarding prevalent behavior is available, people exhibit the tendency to copy this behavior, a phenomena referred to as conformity

• We compare ourselves to others who are better off for guidance, and to others who are worse off to increase our self-esteem

• When outcome information regarding other people’s payoffs is available, people show distributional concerns, such as inequality aversion

MovieLens: method of analysis

• Method: field experiment• Randomized sample of 398 users

with 30 or more reviews• Random money prize assigned to

participants• Divided into 3 experimental groups

• Personal newsletter 1: median number of contributions (of their cohort)

• Personal newsletter 2: net benefit (benefit-cost in time) of average user in own cohort

• Control with their own ratings

• Stage one: pre-experimental survey to estimate benefits in membership

A procedural ordering of methods to develop incentive compatible applications

Any question?

Thank you

Roberta CuelUniversity of Trento & KIT

[email protected]

Insemtives at the KIT

Business

Transcript of Insemtives at the KIT