THE FOUR PILLARS OF

CROWDSOURCING

A Reference Model

Mahmood HosseiniKeith PhalpJacqui TaylorRaian Ali

Faculty of Science and TechnologyBournemouth University, UK

2

OUTLINEIntroduction

Why a taxonomy?

Research method

The four pillars of crowdsourcing

Crosscutting dependencies

Information systems and crowdsourcing

Conclusion

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

3

INTRODUCTION

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

4

CROWDSOURCING DEFINITION The original definition by Jeff Howe

(2006): “Crowdsourcing is the act of taking a job

traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.”

Crowdsourcing has gained much attention since!

A variety of applications A variety of platforms A variety of definitions

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

5

BUT WHAT IS CROWDSOURCING REALLY? No agreed definition

(E. Estellés-Arolas, 2012, “Towards an integrated crowdsourcing definition”)

No established ontological and conceptual foundation

No taxonomy for crowdsourcing

Do we need an agreed definition? Do we need a conceptual foundation? Do we need taxonomy for

crowdsourcing?

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

6

WHY A TAXONOMY?

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

7

WHAT IS A TAXONOMY? Taxonomy brings order to a body of

information or a collection of objects under study.

Taxonomy very often enhances one's ability to understand complex concepts and ideas.

Taxonomy classifies or categorizes a body of knowledge or collection of objects in a hierarchical manner.

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

8

WHY IS A TAXONOMY NEEDED IN CROWDSOURCING? Taxonomies are useful:

Taxonomy of reverse engineering Chikofsky and Cross, “Reverse engineering and

design recovery: a taxonomy”Taxonomy of autonomous agents

Franklin and Graesser, “Is it an agent, or just a program?: a taxonomy for autonomous agents”

Taxonomy of web search Broder, “A taxonomy of web search”

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

9

WHY IS A TAXONOMY NEEDED IN CROWDSOURCING? Crowdsourcing definitions are partial, diverse,

and sometimes even contradicting. Take these examples:

“By ‘crowdsourcing’ I mean: Tapping the perceptual, cognitive or enactive abilities of many people to achieve a well-defined result such as solving a problem, classifying a data set, or producing a decision.” Erickson, 2011

“Crowdsourcing has recently emerged as a powerful alternative. It outsources tasks to a distributed group of people (usually called workers) who might be inexperienced on these tasks.” Li et al., 2013

“A crowdsourcing market is the place, usually an online website, where workers find and perform tasks often for a financial reward.” Fardani et al., 2011

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

10

RESEARCH METHOD

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

11

HOW WE CONDUCTED THIS RESEARCH Literature study in crowdsourcing

To get a better understanding of the concept Multidisciplinary (computer science, business

and management, law, medicine, environmental sciences, etc.)

Papers with an explicit definition of crowdsourcing

Content analysis Extracting crowdsourcing features

Three reviewers involved Several passes to ensure no features are left

unnoticed Building up on our taxonomy

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

12

THE FOUR PILLARS OF CROWDSOURCING

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

13

WHAT ARE THE FOUR PILLARS? The Crowd

These are the people who participate in a crowdsourcing activity

The Crowdsourcer These are the people who are seeking for a

solution to a problem, an innovation, or co-creation

The Crowdsourced Task This is what needs to be performed, solved,

created or innovated by the crowd

The Crowdsourcing Platform This is where it all takes place

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

14

TAXONOMY OF THE CROWD Diversity

Spatial diversity Gender diversity Age diversity Expertise diversity

Unknown-ness Not known to

crowdsourcer Not known to each

other Largeness

Number fulfils the task

Number not abundant

Undefined-ness Suitability

CompetenceCollaborationVolunteeringMotivation

Mental satisfaction Self-esteem Personal skill

development Knowledge sharing Love of

community

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

15

TAXONOMY OF THE CROWDSOURCER Incentives provision

Financial incentivesSocial incentivesEntertainment incentives

Open call Ethicality provision

Opt-out provisionFeedback to crowdNo harm to crowd

Privacy provision

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

16

TAXONOMY OF THE CROWDSOURCED TASK Traditional operation

In-house Outsourced

Outsourcing task Modularity

Atomic tasks Divisible to micro

tasks Complexity

Simple tasks Complex tasks

Solvability Simple for humans Complex for

computers

Automation characteristics Difficult to

automate Expensive to

automate User-driven

Problem solving Innovation Co-creation

Contribution Type Individual

contribution Collaborative

contribution

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

17

TAXONOMY OF THE CROWDSOURCING PLATFORM Crowd-related Interactions

Provide Enrolment Provide Authentication Provide Skill Declaration Provide Task Assignment Provide Assistance Provide Result Submission Coordinate Crowd Supervise Crowd Provide Feedback Loops

Crowdsourcer-related Interactions Provide Enrolment Provide Authentication Provide Task Broadcast Provide Assistance Provide Time Negotiation Provide Price Negotiation Provide Result Verification Provide Feedback Loops

Task-related Facilities Aggregate Results Hide Results from Others Store History of Completed Tasks Provide Quality Threshold Provide Quantity Threshold

Platform-related Facilities Online Environment Manage Platform Misuse Provide Ease of Use Provide Attraction Provide Interaction Provide Payment Mechanism

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

18

CROSS-CUTTING DEPENDENCIES

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

19

WHAT DO WE NEED TO KNOW? There are clearly a set of inter-

dependencies between some crowdsourcing features

They help developers and decision-makers

We use four relationship notionsRequireExcludeSupport (a softer version of require)Hinder (a softer version of exclude)

Some crowdsourcing features are social and qualitative!

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

20

INTER-DEPENDENCIES EXAMPLES Co-creation, a task feature, requires Collaboration,

a crowd feature. Largeness, a crowd feature, supports Diversity, a

crowd feature. It hinders Coordinate Crowd, a platform facility.

Not Known to Each Other, a crowd feature, excludes, Collaboration another crowd feature. It also excludes Collaborative Contribution, a task feature. It hinders Social Incentives, a crowdsourcer feature, as the participants’ identities will be hidden from each other.

Provide Skill Declaration, a platform facility, supports Provide Task Assignment, another platform facility. It hinders Not Known to Crowdsourcer, a crowd feature.

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

21

INFORMATION SYSTEMS AND CROWDSOURCING

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

22

CHALLENGES Metrics

Fuzzy concepts (e.g. largeness) Diversity

Trade-off between quality and diversity Recruitment

Trade-off between quality and open call Interaction and aggregation

Regulations needed because of the intensive human factor

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

23

WHERE CAN WE UTILIZE CROWDSOURCING IN INFORMATION SYSTEMS? Adaptive systems engineering

Ali et al. “Social adaptation: when software gives users a voice”

User feedback for requirements knowledge Pagano and Maalej “User feedback in the appstore: an empirical study”

Stakeholders discovery Lim et al. “StakeNet: using social networks to analyse the stakeholders

of large-scale software projects”

Testing Bell et al. “Secret ninja testing with HALO software engineering”

Requirements elicitation Knauss “On the usage of context for requirements elicitation: end-user

involvement in IT ecosystems”

Validation Sayeed et al. “Crowdsourcing the evaluation of a domain-adapted named

entity recognition system”

More

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

24

CONCLUSION

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

25

CONCLUSION We provide an initial taxonomy for

crowdsourcing The taxonomy will help researchers to

better understand crowdsourcing and its features

The taxonomy will help practitioners to design crowdsourcing platforms more efficiently

The taxonomy will better clarify the usage areas of crowdsourcing in any domain, including information systems

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

26

FUTURE WORK

Taxonomy revision and extension

Capturing the crowd

Developing tools

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

27

ACKNOWLEDGMENT

The research was supported by an FP7 Marie Curie CIG grant (the SOCIAD Project) and by Bournemouth University through the Fusion Investment Fund (the BBB and VolaComp and BUUU projects) and the Graduate School Santander Grant for PGR Development.

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k

28

THANK YOU!

RC

IS, M

arr

ake

ch, M

oro

cco, 2

8-3

0 M

ay,

20

14

mh

oss

ein

i@b

ou

rnem

ou

th.a

c.u

k