Contents lists available at ScienceDirect

Information Processing and Management

journal homepage: www.elsevier.com/locate/infoproman

Understanding crowdsourcing projects: A systematic review oftendencies, workflow, and quality management

Fábio R. Assis Neto⁎, Celso A.S. SantosFederal University of Espirito Santo, Av. Fernando Ferrari, 514 - Goiabeiras, Vitória, ES, 29075-910, Brazil

A R T I C L E I N F O

Keywords:Systematic reviewCrowdsourcingCrowdsourcing projectsQuality managementCrowdsourcing workflow

A B S T R A C T

Crowdsourcing is a powerful method that leverages human intelligence to solve problems in themedical, linguistics, machine learning, and in a considerable number of other fields of study. Thissystematic review aims to understand how crowdsourcing projects are designed and executed inthe state-of-the-art, considering the following dimensions: Task execution, quality management,and platform usage. Our results summarized trends of the important aspects of a crowdsourcingproject, such as crowd and task types, crowdsourcing platforms, and activities used to managethe quality; we also addressed functions and limitations in traditional crowdsourcing platforms,the definition of a crowdsourcing workflow, and the lack of standardization when developing acrowdsourcing project. As future works, we wish to design a conceptual model that represents theimportant elements of a crowdsourcing project and their relationships, also, we wish to do fur-ther researchers focusing on how the quality mechanisms listed in this work should be used incrowdsourcing projects to get reliable results.

1. Introduction

Crowdsourcing, in short, is a general-purpose problem-solving method, which uses a group of participants willing to help solve aproposed problem (Doan, Ramakrishnan, & Halevy, 2011). This method has proven useful in medicine (Dumitrache, Aroyo, Welty,Sips, & Levas, 2013; Foncubierta Rodríguez & Müller, 2012; Irshad et al., 2017; Maier-Hein et al., 2014; Vernez et al., 2017; Zhaiet al., 2013), linguistics (Chowdhury et al., 2014; Enochson & Culbertson, 2015; Lasecki et al., 2013), machine learning (Aigrainet al., 2016; Noronha, Hysen, Zhang, & Gajos, 2011; Roemmele, Archer-McClellan, & Gordon, 2014), software development andtesting (Dietl et al., 2012; Dwarakanath et al., 2015; Fava, Shapiro, Osborn, Schäef, & Whitehead Jr., 2016; Leicht, Knop, Blohm,Müller-Bloch, & Leimeister, 2016; Weidema, López, Nayebaziz, Spanghero, & van der Hoek, 2016), and in a considerable number ofother fields of study.

As stated by Yin, Gray, Suri, and Vaughan (2016), the crowdsourcing platforms hide personal attributes and social characteristicsof participants. In these platforms, the communication usually is only in one direction from the person asking for a solution (therequester or problem owner) to each participant individually (the worker). Hence, participants commonly are abstracted as merely ablack box method to accomplish microtasks associated with the problem solution. However, other authors consider the commu-nication in crowdsourcing as a two-way process, such as stated by Oleson et al. (2011), which asked for feedback from the workers togenerate ground truth data (gold units) in their crowdsourcing project. Furthermore, complex tasks require increased communicationwith the workers. As tasks become more complex, the relationship between the crowdsourcer and workers become more akin tooutsourcing than crowdsourcing (Staffelbach et al., 2015).

https://doi.org/10.1016/j.ipm.2018.03.006Received 18 November 2017; Received in revised form 9 March 2018; Accepted 12 March 2018

⁎ Corresponding author.E-mail address: fabio.ribeiro.neto@gmail.com (F.R. Assis Neto).

Information Processing and Management 54 (2018) 490–506

Available online 17 March 20180306-4573/ © 2018 Elsevier Ltd. All rights reserved.

T

A crowdsourcing process can be viewed as both collaborative or cooperative. Roschelle and Teasley (1995) defined two state-ments that try to demonstrate the difference between collaborative and cooperative work:

Collaboration is a coordinated, synchronous activity that is the result of a continued attempt to construct and maintain a sharedconception of a problem. (p. 70)Cooperation is accomplished by the division of labor among participants as an activity where each person is responsible forsolving a portion of the problem. (p. 70)

In a crowdsourcing collaborative process, collaborators usually have a broad view of the entire process and frequently rely on thecommunication and contributions of other participants to realize their work, e.g., Idea Jam1 and Ushahidi2 approaches. In anotherhand, a crowdsourcing cooperative process usually is controlled by a human computation algorithm, in which workers contributeindividually, and applied to solve a problem proposed by a crowdsourcer and hidden from the workers, e.g., Mturk.

By following a workflow, each member of the crowd works on a portion or on the entire problem, performing a human com-putation task which produces useful results towards the solution to the crowdsourcer problem. Planning, coordination, and control ofprocess are defined by a production manager, usually, the crowdsourcer, although other authors argue that the responsibility forworkflow design can be shared between the crowd and the crowdsourcer (Kulkarni, Can, & Hartmann, 2012).

Kittur et al. (2013) defined that crowdsourcing workflows are needed once they facilitate decomposing tasks into subtasks,managing the dependencies between subtasks, and assembling the results. However, our view of this concept is broader. Crowd-sourcing workflow is context-oriented and should be composed of at least one or more tasks sets and one or more quality controlactivities to guarantee reliable results, which can be executed in parallel, iterative, and sequential ways. Considering our conceptions,it is possible to set quality activities before, during, or after the task execution stages, thus allowing the crowdsourcer to betterunderstand how the quality will be managed during his project. Considering that a workflow can contain multiple task sets, each setis composed of tasks with the same objective, which are executed following a workflow. Thus, our conceptions conform the ideas ofKittur et al. (2013), hence facilitating decomposing a complex task into subtask sets, managing their dependencies, and assemblingtheir results.

Hence, in our conception, a crowdsourcing workflow is a unique entity related to each project. It should represent the planningand coordination done by the crowdsourcer in the entire project, thus, representing a broader view of who will participate, what willbe produced, and how it will be done at every stage of the workflow.

Popular studies in crowdsourcing, as in Estellés-Arolas and González-Ladrón-de Guevara (2012), Brabham (2012), and Zhao andZhu (2014) present the development of crowdsourcing field of study by evaluating known systems and general-purpose platforms, asthe Amazon Mechanical Turk (Mturk),3 CrowdFlower,4 Innocentive,5 and Threadless6.

Classic definitions of the crowdsourcing term usually neglected concepts that we consider crucial for every project, in particularthe definition of a crowdsourcing workflow and methods to control the quality of the entire project. Hence, this issues motivated us toreview the state-of-the-art of crowdsourcing and try to understand how crowdsourcing projects were executed, not only how systemsand platforms works. Thus aiming to validate our assumptions and disseminate new conceptions.

In this review, we extend the work presented in previous crowdsourcing surveys and address essential points to be consideredwhen developing a crowdsourcing project. If popular papers in this domain evaluated crowdsourcing platforms and systems, our viewis broader, regarding the evaluation of quality management and the essential elements of a crowdsourcing project. Furthermore, wepresent classifications and trends regarding the crowd, tasks, platforms, task results, products generated, and the quality managementin crowdsourcing projects. Thus, one of our goals is to highlight the main concerns of crowdsourcing projects. These concepts providecrowdsourcers and researchers with essential information to successfully design their projects. This paper is, therefore, an up-to-datesystematic literature review of crowdsourcing projects that aims to provide detailed information about this new research field. Hence,this review presents the following primary contributions:

• Classifications and tendencies of crowd types, task types, crowdsourcing platforms, and activities used to manage the quality of acrowdsourcing project.

• Observations about the limitations of traditional crowdsourcing platforms.

• Our view of the crowdsourcing workflow, regarding tasks and quality management.

This paper is organized as follows: Section 2 presents state-of-the-art background; Section 3 introduces the methods used in ourapproach; Section 4 presents the results of our review; Section 5 presents a discussion about all information obtained during thereview process. Finally, Section 6 proposes future works and concludes this paper.

1 ideajam.io2 ushahidi.com3 mturk.com4 crowdflower.com5 innocentive.com6 threadless.com

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

491

2. Background

The first report about crowdsourcing related to the information technology field of study was written by Howe (2006), andpublished by Wired Magazine.7 This article presents a comparison between an online tool called IStockPhoto,8 which collects andprovides images using crowdsourcing, to the services offered by a professional photographer.

To define crowdsourcing, Howe (2008) proposed the following description:

Crowdsourcing is the act of taking a task traditionally performed by a designated agent (such as an employee or a contractor) andoutsourcing it by making an open call to an undefined but large group of people. Crowdsourcing allows the power of the crowd toaccomplish tasks that were once the province of just a specialized few. (p. 1)

However, the crowdsourcing field of study grew in popularity, and diverse authors proposed several new definitions for the term.In a renowned publication, Estellés-Arolas and González-Ladrón-de Guevara (2012) reviewed various papers and proposed a unifieddescription:

Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, orcompany proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, thevoluntary undertaking of a task. The undertaking of the task, of variable complexity and modularity, and in which the crowdshould participate bringing their work, money, knowledge and/or experience, always entails mutual benefit. The user will receivethe satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills,while the crowdsourcer will obtain and utilize to their advantage that what the user has brought to the venture, whose form willdepend on the type of activity undertaken. (p. 197)

The work of Hosseini, Phalp, Taylor, and Ali (2014) focused on defining four essential pillars of crowdsourcing. The crowd thatconsists of workers who take part in a crowdsourcing project; the crowdsourcer which is the entity that plans, coordinates, andcontrols the crowdsourcing project; the crowdsourcing task consists of the activities to be solved by the workers; and the fourth pillarrefers to the crowdsourcing platform, which manages the crowd and the tasks.

Considering the proposed definition, Estellés-Arolas and González-Ladrón-de Guevara (2012) formulated eight characteristics andused them for evaluating crowdsourcing systems. These components are listed below:

1. “There is a clearly defined crowd.”2. “There exists a task with a clear goal.”3. “The recompense received by the crowd is clear.”4. “The crowdsourcer is clearly identified.”5. “The compensation to be received by the crowdsourcer is clearly defined.”6. “It is an online assigned process of participative type.”7. “It uses an open call of variable extent.”8. “It uses the Internet.” (p. 197)

Alongside the eight characteristics listed above, Zhao and Zhu (2014) presented a conceptual framework, based on the designquestions behind collective intelligence, proposed by Malone, Laubacher, and Dellarocas (2010), that leverages important concepts inthe crowdsourcing domain. This framework proposes the following key questions to differ this domain from others:

1. “Who is performing the task?”2. “Why are they doing it?”3. “How is the task performed?”4. “What about the ownership and what is being accomplished?”

The first question relates to the composition of a crowd, which can be classified as a group of anonymous people or a group ofspecific people. The second question is about what incentives the workers to contribute to a crowdsourcing project, and according toRyan and Deci (2000), these incentives can be intrinsic and reflect the natural human propensity to learn and assimilate, or they canbe extrinsic and reflect external control or true self-regulation. The third issue relates how the task is performed, whether it will bedivided into microtasks that compound a result by blending all microtasks result (e.g., Mturk), or the task will be realized throughcompetition, selecting and rewarding the best results (e.g., freelancer9). Tasks can also be solved by free contribution, allowing theworkers to use their creativity and define how they can contribute to the project. The last question is about the product generated,that can belong to the public or private domain (Zhao & Zhu, 2014).

Estellés-Arolas, Navarro-Giner, and González-Ladrón-de Guevara (2015) also conducted a literature review searching for differentcrowdsourcing typologies. Then, they stated an integrated typology composed of five main types:

7 wired.com8 istockphoto.com9 freelancer.com

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

492

1. Crowdcasting: Competitive crowdsourcing initiatives, where a problem is proposed to the crowd and the worker who solves it firstor do it better receive a reward.

2. Crowdcollaboration: Crowdsourcing initiatives in which communication occurs between workers of the crowd.3. Crowdcontent: The crowdsourcer seeks crowd labor and knowledge to create or find content of various types but not in a

competitive way.4. Crowdfunding: When an individual or an organization seeks for funding from the crowd.5. Crowdopinion: When the crowdsourcer seeks user opinions and validation about a particular issue or product through votes,

comments, tags, or ratings.

As crowdsourcing is a general-purpose method, a standard issue is how to classify tasks in this field of study. To address this issue,Brabham (2012) defined four categories of crowdsourcing tasks according to what problem the crowdsourcer seeks to solve.Knowledge Discovery and Management consist of tasks that find and collect information into a common location and format; Tasksthat solve empirical problems are classified as Broadcast Search; Peer-Vetted Creative Production organizes tasks that create andselect ideas; Tasks that analyze a significant amount of information are classified as Distributed Human Intelligence Tasks.

Besides the previous categories of Estellés-Arolas et al. (2015), Zhao and Zhu (2014), and Brabham (2012), Geiger, Fielt,Rosemann, and Schader (2012) observed two fundamental dimensions related to crowd participation in crowdsourcing systems. Thefirst dimension consists of systems that seek homogeneous or heterogeneous contributions from the crowd: A homogeneous systemvalues all (valid) contributions equally, i.e., all the inputs are qualitatively equal. In contrast, a heterogeneous system values thecontributions according to their qualities, i.e., all inputs are qualitatively different. While the second classifies systems that seekemergent or non-emergent values from these contributions: A non-emergent system generates its product by selecting individualcontributions, while an emergent system generates its product from the entirety of inputs and the relationship between them, i.e.,combining all contributions and selecting the crowd consensus.

Considering the previous two dimensions, Geiger et al. (2012) categorized the crowdsourcing systems in four types according tothe execution of the crowdsourcing process and how its product is generated.

The Crowd Rating type classifies systems that collect massive homogeneous contribution and seeks to generate their productthrough aggregations and relations between all contributions. The consensus between collaborations provides a collective answer,such as a spectrum of opinions or common assessments and predictions that reflect the wisdom of crowds (Geiger et al., 2012).

Crowd Processing systems collect massive homogeneous contributions and seek non-emergent value that derives directly from theindividual contributions. These systems commonly divide a task into micro tasks and merge all individual contributions of onemicrotask to generate a collective result, e.g., labeling images (Geiger et al., 2012).

Systems that collect heterogeneous contributions and their products derive from aggregations and relationships between allcontributions are classified as Crowd Creation systems. Each input is qualitatively different and aggregating these contributions mustgenerate a comprehensive product, e.g., an image database (Geiger et al., 2012).

Crowd Solving consists of systems that collect heterogeneous contributions and seeks non-emergent value that derives directlyfrom the individual contributions. Specific evaluation criteria define the qualitative value of a contribution. Workers on these systemspresent alternative or complementary solutions to a proposed problem; then the product is generated by evaluating and selecting thebest solutions (Geiger et al., 2012).

In the next section, we appropriately evaluate the information exposed in our state-of-the-art background to define importantconcepts and classifications used for searching, selecting, reviewing, classifying, and gathering data from crowdsourcing papers andprojects.

3. Methods

This section presents methods used in this review for filtering and selecting scientific papers that related crowdsourcing projects.These methods are elaborated based on classifications given by the works cited in the previous section.

3.1. Research questions

This study seeks to understand how crowdsourcing projects are reported by the scientific community. To fulfill this purpose wedesigned the following research questions:

• (RQ1) What are the tendencies related to crowd management, platforms used, and task types in crowdsourcing projects?

• (RQ2) What are the limitations of general-purpose platforms used in crowdsourcing projects?

• (RQ3) How is quality managed in the crowdsourcing projects?

3.2. Selecting crowdsourcing projects

Considering that the characteristics proposed by Estellés-Arolas and González-Ladrón-de Guevara (2012) were defined with thepurpose of analyzing crowdsourcing systems, we compared and contrasted these features with the conceptual framework proposed byZhao and Zhu (2014).

Estellés-Arolas and González-Ladrón-de Guevara (2012) concluded that a crowd is composed of a group of individuals with the

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

493

characteristics of size, heterogeneity, and knowledge determined by each project. To classify the crowd used in each project, wedefined that when the project provides individual information about workers to the crowdsourcer, it uses a specific crowd, whileprojects that don’t provide such information about the workers apply an anonymous crowd.

We did not consider that the use of the Internet in crowdsourcing projects is mandatory because it is possible to use othercommunication means, as we can examine in the work of Gupta, Thies, Cutrell, and Balakrishnan (2012) and in the Ushahidiplatform. These platforms use SMS communication, instead of using the Internet in their crowdsourcing process (Ashley, Corbett,Jones, Garside, & Rambaldi, 2009).

According to previous discussions, we choose to use the four following criteria to filter and select crowdsourcing projects in ourresearch:

1. What is the crowd composition? (Anonymous or Specific)2. What are the incentives offered? (Intrinsic or Extrinsic)3. How the tasks are performed? (Defined Task, Competitive Task, or Free Contribution)4. What product is generated at the end of the project?

Different from Brabham (2012) that seeks to classify crowdsourcing tasks according to the problem being solved, we establishedfour categories of tasks, inspired by the classification of crowdsourcing information systems proposed by Geiger et al. (2012). Fig. 1shows the four categories of crowdsourcing tasks proposed in our approach. This description was used to evidence three essentialmatters in the tasks: (i) What is provided by the crowdsourcer when designing a task; (ii) how the crowd works in a task; and (iii)what is the outcome of a task.

The four types of crowdsourcing task are described as follows:

1. Object Production: The crowdsourcer gives a task description requesting that the workers produce objects, each worker generatesone or more objects, and the outcome is a set of new objects created by the workers.

2. Object for Solution: The crowdsourcer describes a problem and requests workers to generate solutions, the workers individuallyproduce solutions, and the outcome is the best suitable object that solves the given problem.

3. Object Processing: The crowdsourcer gives a set of N objects to be processed by the crowd, the workers process the objectsaccording to the task description, and the outcome can be a set of objects edited by the workers or the given objects alongside a setof new metadata generated by the crowd.

4. Object Evaluation: The crowdsourcer gives a set of N objects to be evaluated by the crowd, the workers assess the objectsaccording to the task description, and the outcome will be the given objects alongside a set of evaluations related to these objects.

Fig. 1. The four types of crowdsourcing tasks.

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

494

We divided these four task types into two dimensions:

1. Generation: This dimension classifies task types that the crowdsourcer gives a task description and request that the workersgenerate objects or solutions for a proposed problem. Each object created in these tasks represents the work of a single worker.

2. Improvement: This dimension classifies task types that the crowdsourcer gives a set of N objects alongside the task description.Each object processed or evaluated in these tasks represents the work of X workers.

3.3. Query string

Aiming to find papers that approach platforms, systems, or applications in crowdsourcing and then highlight articles that focus onplanning and applying crowdsourcing projects, we formulate a query string that includes variations of the term crowdsourcing alongwith systems and platforms that include the terms jobs or tasks. We excluded from our string the terms crowdfunding, crowdsensing,and papers that focus reviews and surveys. Therefore, our query string is defined below:

(crowdsourc*) AND (platform OR software OR application) AND (job OR task)NOT (crowdsensing OR crowdfunding OR survey OR review)This query string was executed on the following digital libraries: (i) ACM Digital Library. (ii) Engineering Village. (iii) IEEE

Xplore. (iv) Science Direct. (v) Scopus. These libraries were selected because of their relevance to the field of computer science, andtheir content is accessible to the authors of this study.

Papers are selected as the result of a match between the query string and their title/abstract/keywords, this research was realizeduntil Oct. 2016. The query string was adapted according to the requirements of each digital library.

3.4. Inclusion and exclusion criteria

Papers that fit the following criteria were selected to be reviewed:

• (INC1) Papers that mainly focus crowdsourcing.Rationale: A considerable number of articles use crowdsourcing to compare results or to complement purposes on other fields ofstudy.

• (INC2) Papers that present planning and execution of crowdsourcing projects.Rationale: Evaluating crowdsourcing projects allows a broad view of the crowdsourcing process.

• (INC3) Papers that were written in English or Portuguese.Rationale: For reasons of feasibility, articles written in another language were excluded.

Papers that fit the following criteria were excluded:

• (EXC1) Short Papers.Rationale: We considered that the papers should have at least four pages to be suitable for our review process.

• (EXC2) The four criteria presented in the Section 3.2.Rationale: Considering that these criteria are essential to identify a crowdsourcing project, we excluded papers that did notpresent sufficient information to report each of them.

• (EXC3) Crowdsourcing projects inexistent or poorly defined.Rationale: The primary focus of our study is understand the crowdsourcing process as a whole. Therefore, it’s essential to evaluatewell-defined projects.

• (EXC4) Similar papers.Rationale: When two or more papers have similar authors, crowdsourcing projects, and results, we included the most detailedarticle.

3.5. Selecting papers

Papers were selected and organized using the Mendeley10 tool, that was chosen because it’s useful for organizing, annotating andfiltering papers. After executing our query string on the selected digital libraries and removing duplicated papers, call for papers, andproceeding descriptions we started our study with the following numbers:

Table 1 introduces the 947 papers that composed our first stage of reviews, which consisted of comparing the defined inclusionand exclusion criteria with the information obtained by reading the title, abstract, and keywords of selected papers. This stageresulted in 813 excluded papers and 133 papers selected for the second stage of reviews, which applies inclusion and exclusioncriteria on the information obtained reading the articles entirely.

The second stage of reviews resulted in 62 selected papers that presented 66 crowdsourcing projects for further analysis. It isnoteworthy that some papers presented multiple crowdsourcing projects, and we considered all the projects that offer different

10 mendeley.com

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

495

classifications.The entire searching and selecting papers process was repeated in May 2017 to update our study and tendencies. Disregarding all

papers found in our first search process, removing duplicates, and call for papers, our second search process found 153 new articles.After applying inclusion and exclusion criteria in the title, abstract, and keywords, 31 of 153 new papers were selected for the

second stage of reviews. In the second stage of reviews, we read and evaluated the 31 articles, and after this step, we selected tenpapers that presented well-defined crowdsourcing projects to include in our study. We also published all references found in ourquery process in a Mendeley Dataset.11

Unifying the two search process mentioned above, we gathered and evaluated data of 76 crowdsourcing projects found in 72papers.

3.6. Data gathering

To gather relevant data about the selected papers, we classified the crowdsourcing projects using the four criteria mentioned inSection 3.2 plus the three following categories:

1. What is the purpose of the platform used in the project? (General-purpose or Specific-purpose)2. What type of crowdsourcing task does the project apply? (Object processing, Object creation, Object evaluation, Object for

solution)3. What activities are responsible for managing quality during the crowdsourcing process?

After gathering data by answering the questions addressed above in the 76 crowdsourcing projects, we observed tendencies andclassifications that are presented in the next section.

4. Results

4.1. What is the crowd composition?

Our study indicated 50 crowdsourcing projects that the crowdsourcer doesn’t store information about the workers, classified as ananonymous crowd, and 26 projects in which the crowdsourcer stores individual information about the workers, classified as a specificcrowd.

Regarding the crowdsourcing projects that approached an anonymous crowd, 74% used general-purpose platforms (e.g. Mturkand CrowdFlower). However, 73,07% of the projects that approached a specific crowd used specific-purpose platforms.

It’s noticeable that general-purpose platforms are likely used for managing an anonymous crowd. General-purpose platformstypically manage several crowdsourcing projects, distributing tasks to anonymous workers and collecting results. If a crowdsourcerwants to store information about the workers, it’s possible to control a crowd by developing a specific platform that supplies hisneeds, defining then a specific-purpose platform.

Table 2 indicates that most of the crowdsourcing projects that manage an anonymous crowd seek to perform object processing(e.g. image segmentation, video annotation, text recognition). Therefore, the lack of information about the workers, that defines thepseudo-anonymity of the crowd (Chandler & Kapelner, 2013; Chowdhury et al., 2014; Kim, Li, chul Kwon, & Yi, 2011), limits theexecution of complex tasks by anonymous workers. Lack of information about the crowd hinders crowd management, qualitymanagement, and evidence demographic limitations (Kim et al., 2011).

Regarding the projects that approach specific crowds, even if 69,23% of the cases seek to perform object processing, studiessimilar to Feng et al. (2016) are viable. Their study presents the development of an educational game that aims to provide knowledgeto students while they analyze biomedical images. Specific crowds also allow projects to be performed in laboratories, universitycampus, and specific geographic locations, as presented in the following papers: (Figuerola Salas, Adzic, Kalva, & Shah, 2013; Guptaet al., 2012; Kermanidis, Maragoudakis, & Vosinakis, 2015; Lasecki et al., 2012; Tuite, Snavely, Hsiao, Tabing, & Popović, 2011).

Storing information about workers enables the management of a crowd composed of specialists in specific knowledge fields, as

Table 1Papers found in the digital libraries.

Digital library Number of papers

ACM Digital Library 378Engineering Village 268IEEE Xplore 249Science Direct 50Scopus 698Total excluding intersections 947

11 https://doi.org/10.17632/z5mrss39v7.1

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

496

approached in the studies of (Chowdhury et al., 2014; Dumitrache et al., 2013; Dwarakanath et al., 2015; Foncubierta Rodríguez &Müller, 2012; Huynh, Eguchi, Lin, & Eguchi, 2014; McAllister Byun, Halpin, & Szeredi, 2015). It’s also possible to assemble users fromplatforms that aren’t specifically developed for crowdsourcing means, such as the Facebook12 and Adobe Photoshop,13 presented inthe following studies: (Banks, Rafter, & Smyth, 2015; Dontcheva, Morris, Brandt, & Gerber, 2014; Sabou, Scharl, & Föls, 2013).

4.2. What are the incentives offered?

Hosseini, Shahri, Phalp, Taylor, and Ali (2015) stated that incentives can be intrinsic or extrinsic. Intrinsic motivations are afeature within the crowd, while the extrinsic ones come from the crowdsourcer, in the form of financial incentives or other forms.They also identified five types of intrinsic incentives: (i) Knowledge sharing; (ii) Love of community; (iii) Personal skills development;(iv) Self-esteem; and (v) Volunteering. During our classification, we summarized the incentives found within categories that we foundrelevant to calculate our tendencies.

We observed that 51 of 76 reviewed projects presented extrinsic incentives, while 21 projects used intrinsic incentives and fourprojects approached multiples motivations, that incentivized workers intrinsically and extrinsically.

Considering the information shown in Table 3, it’s noticeable the predominance of monetary incentives, such as payments foreach task done, raffles, and rewards to the best contributors. Financial incentives, mainly used in commercial platforms such as theMturk, have the advantage of attracting large amounts of people to compose a crowd (Cao, Chen, & Jagadish, 2014; Kim, Kim, Yang,Ryu, & Wohn, 2014; Kim et al., 2011). Nevertheless, monetary incentives can be harmful to a crowdsourcing project, attractingmalicious workers that only seeks to obtain financial rewards without caring about the work done (Cao et al., 2014; Kim et al., 2014).Bénabou and Tirole (2003) stated that monetary rewards also can undermine intrinsic incentives and become negative reinforcersonce they are removed.

Access incentives are presented in papers which use the CAPTCHA approach (von Ahn, 2005). This method aims to preventautomatic access to websites by giving tests that only humans can solve. Using access as crowdsourcing incentive means that theworkers will need to solve tasks in exchange to access specific sites. By solving these tasks the workers are contributing to acrowdsourcing project, as presented in the studies of Law and Ahn (2011), Hillen and Höfle (2015), and Kim et al. (2014).

Another extrinsic incentive is presented in the work of Feng et al. (2016), which motivates workers by providing school suppliesto the students that participated in their crowdsourcing experiment.

Table 4 shows that 60% of the projects that use intrinsic incentives approached entertainment as motivation for the crowd. Thisincentive is defined by the term Games-With-a-Purpose (GWAP) attributed by von Ahn and Dabbish (2008), which addresses gamesdeveloped with the purpose of collecting human intelligence through tasks that entertain the player (von Ahn & Dabbish, 2008; Tuiteet al., 2011).

Social incentives are exemplified in the study of Vukovic, Salapura, and Rajagopal (2013), which publicly acknowledge the workdone in the crowdsourcing tasks. Considering that the workers in this project belong to the same type of business, acquiring ac-knowledgment in this group is a viable incentive to execute tasks.

Projects that use altruism as incentive seek to mobilize people to compose a crowd and work to benefit someone else. Studiesfound include natural disasters, as in the work of Hester, Shaw, and Biewald (2010), which describes the effort to categorize andlocate information received by SMS sent from people in an emergency during the earthquake that happened in Haiti, 2010. In thissame natural disaster, the study of Zhai, Kijewski-Correa, Hachen, and Madey (2012) seeks to identify damage done in buildings andstructures. The work of Huynh et al. (2014) has the same objective as the study of Zhai et al., but was executed during the earth-quakes and tsunamis that happened in Japan, 2011.

Learning incentives are exemplified by the study of Dontcheva et al. (2014), which presents a crowdsourcing platform builtaround the Adobe Photoshop platform. This crowdsourcing system offers advanced tutorials of image editing and provides trainingtasks to the workers, and then it’s possible to generate a database of edited images by combining the results of these tasks.

The studies of Feng et al. (2016) and Tuite et al. (2011) exemplifies the multiple incentives approach, by presenting entertainmentincentives as the primary incentive and offering extrinsic incentives, school supplies, and financial incentives respectively, to the bestcontributors.

4.3. How the tasks are performed?

Observing the crowdsourcing projects reviewed, 71 used the defined task model, four cases approached the free contribution

Table 2Tendencies in projects featuring anonymous and specific crowds.

Crowd members Object processing Object production Object evaluation Object for solution Total

Anonymous 41 (82%) 2 (4%) 6 (12%) 1 (2%) 50 (100%)Specific 18 (69,23%) 5 (19,23%) 2 (7,69%) 1 (3,85%) 26 (100%)

12 facebook.com13 adobe.com/photoshop

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

497

model, and one focused on the competition method.Table 5 presents the processing tasks as leading trends, which consists of tasks that request the workers to evaluate an object and

produce something, like annotations and markings. Object processing includes combining an object provided by the crowdsourcerwith task results produced by workers, thus generating new objects or metadata to complement given objects. This approach wasdetailed in Section 3.2.

Image processing is exemplified in studies that produced annotations (Foncubierta Rodríguez & Müller, 2012; Redi et al., 2013) ormarkings (Della Mea, Maddalena, Mizzaro, Machin, & Beltrami, 2014; Estes et al., 2016). Marking images differs from annotatingimages because the markings are grouped to the visual content of an image, while annotations are independent because they useinformation contained in the image to generate additional content, such as metadata and text transcription.

Regarding video processing, the work of Sulser, Giangreco, and Schuldt (2014), and Nguyen-Dinh, Waldburger, Roggen, andTröster (2013) approached video annotation according to events that occurred during the video. The crowdsourcing project pre-sented in Masiar and Simko (2015) seeks to collect video metadata by using the crowd to search and compare the search results. InDi Salvo, Spampinato, and Giordano (2016) workers segmented videos by playing a game about marine species. Figuerola Salaset al. (2013) uses human intelligence to compare video quality.

As for text processing, it’s noticeable in the studies of Zhai et al. (2013) and Dumitrache et al. (2013) the usefulness of crowd-sourcing to recognize medical text and combine them to generate reliable databases. The work of Mellebeek et al. (2010) appeals tothe crowd for evaluating opinions about car brands, seeking to train an opinion mining system.

Evaluating tasks that produce audio processing, the work of Lasecki et al. (2012) presents an approach for transcribing andsubtitling audios on demand. The study of McAllister Byun et al. (2015) proposes to evaluate speech using crowdsourcing,McNaney et al. (2016) describes a mobile app for people with Parkinson’s condition to manage their dialogue, receiving feedbackgenerated by a crowdsourcing project.

Crowdsourcing is a versatile approach when considering tasks and their results, therefore, a remarkable number of reviewedprojects differ from the classifications presented above, representing then the others category with 23% of the sampling exposed onTable 5.

Among the others category, three projects used crowdsourcing to verify software. Weidema et al. (2016) formulated tasks that aimto collect alternative design solutions to solve user interface problems, while in Pastore, Mariani, and Fraser (2013) is proposed toevaluate the source code and behavior of a program, aiming to find bugs in the software.

The crowdsourcing project presented in To, Geraldes, Shahabi, Kim, and Prendinger (2016) provides an approach that theworkers should visit selected landmarks, in a defined geographic region, and register how the landmark influences their mood. Thisproject generates as a product one map with landmarks and the feeling related with each.

Table 3Incidence of extrinsic incentives.

Incentive Incidence

Access 02 (03,64%)Monetary 51 (92,73%)Others 02 (03,64%)Total 55 (100,00%)

Table 4Incidence of intrinsic incentives.

Incentive Incidence

Altruism 05 (20%)Entertainment 15 (60%)Learning 02 (08%)Social 03 (12%)Total 25 (100%)

Table 5Tendencies of the defined tasks category.

Defined tasks Incidence

Audio processing 06 (08%)Image processing 32 (45%)Text processing 08 (11%)Video processing 09 (13%)Others 16 (23%)total 71 (100%)

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

498

Vukovic et al. (2013) developed a project that uses the intelligence of a specific crowd to compose technical documents. The workof Banks et al. (2015) proposed a game developed on the Facebook platform, that allows the users to recommend movies to theirfriends, collecting recommendation data afterwards.

Free contribution tasks allow the crowd to use their creativity during the crowdsourcing process. The studies of Tuite et al. (2011)and Bayas et al. (2016) allow the workers to take and send pictures related to a defined locale.

Lasecki et al. (2013) requests that workers answer questions in a chat system with their own words. All workers of this project willvote, and the answer with most votes will be shown to the user that asked the question.

In competitive tasks, as the name suggests, the workers compete by solving and submitting tasks to the crowdsourcer, who selectsand reward the solution that best fits his requests, as approached in the software development website TopCoder14.Dwarakanath et al. (2015) presented a methodology that manages a software development lifecycle, with the code implementationpart produced with competitive tasks. Competition is useful to find specialists capable of producing reliable and complex results.However, skilled workers that work in competitive crowdsourcing platforms frequently seeks better monetary rewards than thoseoffered by microtask platforms (Dontcheva et al., 2014).

The second dimension presented in the work of Geiger, Seedorf, Schulze, Nickerson, and Schader (2011) can be an interesting wayto classify how crowdsourcing tasks are performed. This dimension, called accessibility of peer contributions, indicates if workers canaccess each other’s contributions. The four characteristics of this dimension are: (i) None, which means that workers cannot see eachother’s contributions; (ii) View, in which all contributions are visible to any potential worker participating in the project; (iii) Assess,in which workers can vote, rate, and comment on other contributions; lastly, the (iv) Modify characteristic, which allows workers tomodify and even delete each other’s contributions.

4.4. What product is generated at the end of the project?

Table 6 shows the classification of products generated by crowdsourcing projects, pointing up that crowdsourcing product candiffer to the result of a paper. This statement is important because crowdsourcing is considerably used to produce media databasesand these databases are used to support systems and train artificial intelligence, as presented in Altmeyer, Lessel, and Krüger (2016),Maier-Hein et al. (2014).

Verification of systems is exemplified in the works of Dietl et al. (2012) and Fava et al. (2016), which proposes to verify systemsby using crowd intelligence in GWAP.

The study of Lasecki et al. (2013) presents a chat system powered by crowdsourcing. The workers in this project are monetarilyrewarded by voting on the best answer or by producing the most voted answer.

The work of Tuite et al. (2011) developed a competitive game that motivated University of Washington students to take picturesof defined buildings in the university. As a result of this project, 3D building models were generated from the pictures taken by theworkers, available at the Photo City website.15

4.5. What is the purpose of the platform used in the project?

This study focuses on understanding how the crowdsourcing process is executed nowadays, therefore, is important to evaluategeneral-purpose platforms and their limitations. Reviewing crowdsourcing projects allowed us to perceive the positive and negativesides of these platforms, due to papers that developed specific-purpose platforms to execute crowdsourcing projects, which frequentlydescribed the limitations in general-purpose platforms that led them to develop a specific platform.

Among the 76 reviewed projects, 44 of them were managed by general-purpose crowdsourcing platforms. Table 7 illustrates thatthe Mturk platform shows greater popularity between these platforms. This platform is capable of managing the development,execution, and validation of crowdsourcing tasks quickly and at low cost, due to numerous subscribed workers available to solve tasks(Alonso & Mizzaro, 2012; McAllister Byun et al., 2015; Yang et al., 2010).

The Mturk allows the crowdsourcer to manage crowd workers, enabling them to limit worker participation based on previouswork done and acceptance, also based on geographic location of the workers, and by designing qualification tests that the workersshould pass to be allowed to work in a project (Alonso & Mizzaro, 2012; Can, Odobez, & Gatica-Perez, 2014; Chandler & Kapelner,2013; McAllister Byun et al., 2015; Yang et al., 2010). However, our study points out the following limitations when managing acrowd on this platform: (i) Lack of information and control about the workers (Chandler & Kapelner, 2013; Chowdhury et al., 2014;Kim et al., 2011); (ii) crowd participation is geographically limited (Chowdhury et al., 2014; Della Mea et al., 2014; Ross, Zaldivar,Irani, & Tomlinson, 2010); (iii) worker’s incentive is inflexible (Cao et al., 2014; Kim et al., 2014).

Another limitation found on the Mturk is the difficulty in assembling some tasks, forcing crowdsourcers to host their applicationon a separate server and post a link in the tasks (Kim et al., 2011; McAllister Byun et al., 2015). Mturk is also limited in thequalification tests and validation mechanisms provided (Alonso & Mizzaro, 2012; Ashikawa, Kawamura, & Ohsuga, 2015; Kim et al.,2011).

The CrowdFlower platform differs from Mturk in two main points, the first is a possibility to use a proper markup language alongwith HTML, CSS, and Javascript to develop tasks in this platform (Della Mea et al., 2014; Foncubierta Rodríguez & Müller, 2012). The

14 topcoder.com15 photocitygame.com

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

499

second point is being able to aggregate diverse crowdsourcing platforms as task channels, including Mturk, thus allowing thecrowdsourcer to select in what channels his task will be published (Figuerola Salas et al., 2013; Foncubierta Rodríguez & Müller,2012).

Unexpectedly, the Facebook social network was used as a general-purpose platform. This platform provides an API for devel-opment of games and applications, besides providing information about potential workers and their friends. The statements aboveallows crowdsourcers to manage a crowdsourcing project on this platform, using entertainment as intrinsic incentive (Banks et al.,2015; Sabou et al., 2013). It’s noteworthy that the Facebook platform can be used to collect crowdsourcing votes through its commentsystem.

The Freelancer platform presents the competitive crowdsourcing approach (Dwarakanath et al., 2015). However, the platformdeveloped by Brambilla, Ceri, Mauri, and Volonterio (2014) offers an approach that allows the crowdsourcer to modify the executionof a crowdsourcing project by changing between anonymous and specific crowd during the process.

Microworkers (Redi et al., 2013; Sulser et al., 2014), C-SATS (Vernez et al., 2017), Crowdtesting (Leicht et al., 2016), and oDesk(Leano, Wang, & Sarma, 2016) are other general-purpose platforms used in the reviewed projects.

Platforms developed to manage a specific experiment were considered as specific-purpose platforms, presented in 32 of the 76cases.

Specific-purpose platforms were eventually developed to supply specific limitations of general-purpose platforms, as presented inthe studies of Brambilla et al. (2014) and Castano, Ferrara, Genta, and Montanelli (2016), which developed platforms that allow aflexible crowd management. Another important point is that 62,5% of projects that used specific platforms approached intrinsicincentives, while only 11,36% of projects in general-purpose platforms used them as motivation.

Considering projects in which the crowdsourcer stored information about the workers, 59,37% of projects in specific platformmanaged a specific crowd, while only 15,90% of projects in general-purpose platforms approached the specific crowd model.

Table 8 shows that crowdsourcing is mainly used to process objects, disregarding the platform used. The table above also informsthat general-purpose platforms are prone to execute object evaluation and problem-solving approaches, while object production taskswere only performed in specific-purpose platforms.

Some of the reviewed projects presented the development of a crowdsourcing platform that seeks manage tasks and supplylimitations, found in general-purpose platforms, but still uses these systems to select and control the crowd due to the abundantnumber of workers subscribed on these platforms. General-purpose platforms allow the crowdsourcer to design the called externalHITs, which consists of external tasks hosted on another platform, that are provided to the workers through links and access codes, aspresented in the studies of McAllister Byun et al. (2015), and Nguyen-Dinh et al. (2013).

Table 7Usage of general and specific-purpose crowdsourcing platforms.

General-purpose platforms Incidence

Amazon Mechanical Turk 24 (32%)CrowdFlower 11 (14%)Others 09 (12%)Total General-purpose 44 (58%)Total Specific-purpose 32 (42%)Total 76 (100%)

Table 8The incidence of task types in crowdsourcing platform types.

Platform type Object processing Object production Object evaluation Object for solution Total

General 36 (82%) 0 (0%) 6 (14%) 2 (5%) 44 (100%)Specific 23 (72%) 7 (22%) 2 (6%) 0 (0%) 32 (100%)

Table 6Products generated by crowdsourcing projects.

Product Incidence

Processed Audio 06 (08%)Processed Images 34 (45%)Processed Text 07 (09%)Processed Videos 09 (12%)Software Review 06 (08%)Others 14 (18%)Total 76 (100%)

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

500

4.6. Quality management in crowdsourcing

Crowdsourcing projects frequently have problems with guaranteeing reliable results to the crowdsourcer (Kim et al., 2011),including cases that provide monetary rewards (Altmeyer et al., 2016; Cao et al., 2014). Therefore, to guarantee quality in thecrowdsourcing project and avoid wasting resources, the use of mechanisms to control the quality of a crowdsourcing project isnecessary.

Only 10 of 76 crowdsourcing projects reviewed didn’t specify how the quality is managed in their crowdsourcing process. Byanalyzing the quality management of the 66 remaining studies, we observed the following tendencies: (i) Activities that occur beforethe crowdsourcing process (Pre-task); (ii) techniques applied during the task execution (During-task); (iii) mechanisms used after thetask execution (Post-task).

Table 9 presents the activities used in the pre-task stage, present in 34 of 76 projects, and they are responsible for training andfiltering workers, simplifying task execution, and preprocessing objects used to create tasks. However, these methods don’t grant thequality of a product generated by crowdsourcing, because these activities occur before task execution and it’s impossible to predicthow the workers will solve the task. Therefore, these techniques help to avoid malicious workers and poorly done work (Alonso &Mizzaro, 2012; Chowdhury et al., 2014; Nguyen-Dinh et al., 2013; Tavares, Mourão, & Magalhaes, 2013).

A way to qualify workers is to insert a mandatory training stage before task execution. This stage normally consists of a tutorial orvideo that aims to instruct and train workers for executing the crowdsourcing tasks in a project, as presented in the following studies(Dontcheva et al., 2014; Feng et al., 2016; Huynh et al., 2014; Lasecki et al., 2013).

Qualification tests mean to insert a mandatory test that a worker must pass to work on a project. The result of this test defines theability of the workers to solve the proposed tasks, as approached in Estes et al. (2016), Irshad et al. (2015), Weidema et al. (2016).

Preprocessing consists of the use of algorithms and procedures in the objects provided by the crowdsourcer, aiming to design tasksthat can be easily solved by the workers. The work of Dumitrache et al. (2013) preprocesses the objects to add metadata and generateautomatic annotations, thus simplifying the tasks provided to the crowd. Soleymani (2015) preprocessed an image database, re-moving content that is irrelevant to his project.

Worker accuracy is a technique used mainly in general-purpose platforms, because these platforms store execution data ofsubscribed workers, considering the acceptance of previous work done and accounting for a percentage of approval. This mechanismlimits the participation in a project by requiring that the workers should have a defined or higher rate of acceptance (Ashikawa et al.,2015; Can et al., 2014; Yang et al., 2010).

During-task quality management was used in 21 of 76 projects, and 52% of these projects presented a method called task design,which consists of rules inserted at the moment of task execution, aiming to exclude task results that diverge from defined standards.Della Mea et al. (2014) defined that the workers should recognize a minimum number of cells in each image to be considered as avalid task. The work of Zhai et al. (2012) verified the time that a worker spent on a task and compares this time with the average timespent by other workers, thus defining if the worker is executing the task with proper care.

The mechanism called gold standard was used in 62% of the projects that approached during-task quality management, and thistechnique merges ground truth questions with crowdsourcing tasks. Workers that gives the wrong answers to the ground truthquestions were considered unable to respond to a crowdsourcing task correctly, and their results are discarded. This technique wasused in the following studies (Hillen & Höfle, 2015; Tavares et al., 2013; Zhai et al., 2013).

Table 10 presents the post-task quality management, used in 57 of 76 projects. These methods are capable of unifying multipletask results and generates a reliable result and also filter undesirable task results after the task execution.

The majority decision, or majority voting, is the most used activity for quality management, present in 41 crowdsourcing projects.

Table 9Incidence of pre-task mechanisms.

Pre-task mechanisms Incidence

Pre-Processing 11Qualification Tests 12Training 15Worker Accuracy 08Total Incidence 34

Table 10Incidence of post-task mechanisms.

Post-task mechanisms Incidence

Majority Decision 41Post-Processing 11Review 12Subtasks 06Total Incidence 57

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

501

This technique is frequently used in the microtask approach and consist in the replication of one task in a defined number of identictasks. After the replicated tasks are solved, the results are aggregated, and the crowd consensus is considered as the task results, aspresented in the studies of Irshad et al. (2015), Lasecki et al. (2012), Loni et al. (2014), Tuite et al. (2011), Sabou et al. (2013).

The review method consists of manually reviewing the work done by the crowd, thus deciding the acceptance of the work. Thisactivity is approached in the following papers: (Dontcheva et al., 2014; Dumitrache et al., 2013; Lasecki et al., 2012).

The Post-processing technique consists of optimizing the results obtained in a crowdsourcing project, usually with computationalaid. The studies of Hillen and Höfle (2015) and Taborsky, Allen, Blanton, Jain, and Klare (2015) used post-processing to combine theimage markups provided by the crowd. Tuite et al. (2011) unified images sent by the crowd to generate 3D models of buildings.

The subtasks mechanism is used to evaluate work done by a crowd. This technique consists of generating new crowdsourcing tasksand request workers to verify objects produced by a crowdsourcing project, as presented in Roemmele et al. (2014),Ramchurn, Huynh, Venanzi, and Shi (2013).

5. Discussion

This section presents the contribution of our study to the state-of-the-art in crowdsourcing, and important points when planning aproject in this field of study. Therefore, we related the information obtained by reviewing the 72 papers approached in this study withthe four pillars of crowdsourcing presented in Hosseini et al. (2014).

Considering that the crowdsourcer, defined as the first crowdsourcing pillar, manages each project in a unique way, we did nothave sufficient information to discuss this entity. Therefore, in the following sections, we discuss issues regarding the crowd, thetasks, and the platform of the reported crowdsourcing projects.

5.1. About the crowd

The crowdsourcer should understand what the crowd will produce in his project. This statement is essential to know if the workerswill need specific skills to solve the tasks and if these skills can be taught to unspecialized workers. Dontcheva et al. (2014) applied amandatory training stage that taught image editing skills to the workers before applying their crowdsourcing tasks. If the necessaryskills are too complex to be taught, the crowdsourcer needs to hire workers that have previous knowledge requested to solve thetasks. Dwarakanath et al. (2015) hired workers that already have software development skills.

A common concept of economics is that individuals respond to incentives (Bénabou & Tirole, 2003). Thus, it is also important thatthe crowdsourcer defines what incentives will be provided to the workers. The incentives offered influences directly in the worker’sinterest to participate in a project because most everyone who works with others is concerned with motivation, facing the question ofhow much motivation those others, or oneself, has for a task (Ryan & Deci, 2000).

Section 4.2 presents the incentives used in the crowdsourcing projects reviewed in our study. The costs of monetary rewards andvolume of contributions are not approached in our review because this information was not present in all of the reviewed projects.

5.2. About the crowdsourcing task

As aforementioned, crowdsourcing tasks must be linked with incentive mechanisms, which should reward workers according tothe incentives established by the crowdsourcer.

The crowdsourcer should pay attention if the designed task requires assets, which are resources needed to solve one task. Theproject reported in Dontcheva et al. (2014) defined that only workers with access to the Adobe Photoshop platform could participatein it. This example shows that is important to understand if a task will require assets because these can limit worker participation in aproject.

Gupta et al. (2012) evidenced the importance to associate assets needed with a task. This study selected workers that didn’t haveaccess to computers, smartphones, and the Internet, that are assets required to participate in most of the crowdsourcing projects. Thisproject managed and distributed crowdsourcing tasks via SMS and observed that some workers don’t have access to the standardassets needed to solve crowdsourcing tasks. The crowdsourcer of this project found potential workers that cannot subscribe togeneral-purpose platforms.

An important concept is that a project should define activities to manage quality in the crowdsourcing process, which werepresented in Section 4.6 of this review. The management of quality in the task execution process is critical for saving resources andguaranteeing quality in the final product.

By reviewing crowdsourcing projects, we observed that the crowdsourcer must define how the tasks and quality managementactivities will interact, thus designing a workflow for the project.

We also perceived that the dominant crowdsourcing infrastructure is the workflow, which decomposes complex goals into smalltasks (Retelny, Bernstein, & Valentine, 2017). Zhang et al. (2012) defined that a workflow should manage the inputs and outputs ofmultiple simple task modules, and the union of these modules will result in the product of a complex task. Furthermore,Hetmank (2013) determined that a crowdsourcing workflow must manage latency, work price, worker quality, and contributionquality.

Considering the definitions above and reviewing the crowdsourcing projects, we defined that a project workflow is unique andcontext-oriented. Therefore, it is possible to define that all crowdsourcing workflows must have one or more task module and one ormore quality management module. Otherwise, it’s impossible to generate reliable results. To exemplify the execution of the workflow

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

502

of a crowdsourcing project, Fig. 2 shows the workflow described in the study of Lasecki et al. (2013), however adjusting it to theworkflow model proposed in our study.

The project presented in Lasecki et al. (2013) proposed a chat system that seeks to answer questions by crowdsourcing means. Inthis workflow, the workers initially completed a training stage; then the crowd executed the first crowdsourcing task, which consistedof proposing an answer to a question made in the chat system; the second crowdsourcing task consists of evaluating and voting thesuggestions made by other workers in the first task. In the end, the majority decision activity selected the proposal with most votes toanswer the question made in the chat system.

Regarding the creation of a crowdsourcing workflow, Kulkarni et al. (2012) presented an alternative method to divide a complexproblem into small tasks, thus generating a crowdsourcing workflow.

Retelny et al. (2017) suggest that if a complex problem can’t be adapted to the static crowdsourcing workflow approach, it willremain as a fundamental limitation of workflow-based crowdsourcing infrastructures. However, Fig. 1 shows the Object for Solution,which is an alternative crowdsourcing task type capable of solving complex problems, as presented in Section 4.3.

It’s worth pointing out that the second task of the workflow presented above was initially considered as a quality managementactivity, named in the Section 4.6 as subtasks. This second task still manages the quality of the first task results, but after defining thecrowdsourcing workflow, we decided that the subtasks are better represented as a crowdsourcing task of object evaluation type.

5.3. About the crowdsourcing platform

The crowdsourcer must know the crowdsourcing platforms that are available to execute his project. These platforms, the ten-dencies of usage, and their limitations were presented in Section 4.5 of this study.

We also defined that the crowdsourcing platform must execute two primary functions: (i) Worker management and (ii) taskmanagement. We describe these features below:

1. Worker management: Function responsible for recruiting workers according to the criteria defined in a project, realizing filteringsand forwarding the selected workers to the task management function. This feature is also responsible for distributing the rewardsto the workers when these are extrinsic.

2. Task management: Function responsible for distributing crowdsourcing task to the workers and collect the task results.

It’s noteworthy that a single platform can execute these two features, as frequently done in projects that use the Mturk platform.However, distinct platforms can perform these functions, as approached by the CrowdFlower platform, which executes the taskmanagement while outsources the worker management to other general-purpose platforms, including the Mturk platform. Projectsthat use distinct platforms to run these two features are presented in the following studies: (McAllister Byun et al., 2015; Nguyen-Dinh et al., 2013).

6. Conclusion

This paper presented a systematic review of indexed papers, which aims to collect information and define characteristics aboutcrowdsourcing projects and how their results are reported in the literature. Evaluating 76 crowdsourcing projects found in 72 articles,allowed us to broaden our view of the crowdsourcing field of study.

The tendencies and definitions revealed aim to help researchers and crowdsourcers to understand the state-of-the-art ofcrowdsourcing. These contributions are beneficial as the main concerns when designing further crowdsourcing projects.

Regarding all projects reviewed, we also observed that the concept of a crowdsourcing workflow in a project is not widespread.Thus, we found that the reviewed papers described and modeled their crowdsourcing projects and workflows in the way that theyconsidered feasible. Therefore, this lack of standardization aroused our interest, as a future work, to update the definition ofcrowdsourcing projects and create a conceptual model that leverages the concept of a crowdsourcing workflow.

Fig. 2. A common workflow of a crowdsourcing project.

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

503

As another future work, we also wish to evaluate the quality management in crowdsourcing projects better, aiming to build aguideline of how the quality mechanisms cited should be used in each crowdsourcing project to get reliable results.

The limitations of our review are similar to the limitations of other systematic reviews as well. By considering indexed papers,some important material that could contain valid crowdsourcing projects was not included in this review, such as dissertations, whitepapers, short papers, and papers that we did not have access. Also, crowdsourcing projects with commercial purposes can havedifferent configurations from crowdsourcing projects with academic purposes, thus, our point-of-view in this review is limited tocrowdsourcing projects published in the literature.

Acknowledgments

The first author acknowledges FAPES (Process: 74010158) for the financial grant. Celso A. Saibel Santos is currently funded byCNPq (# 312148/2014-3) and FAPES (# 67927378/2015). Our gratitude goes to Ricardo de Almeida Falbo, who made importantcontributions to define the data gathering protocols used in our Review Process. Special thanks to Edmilson Geraldo Balbino Silva,who reviewed our English writing.

Supplementary material

Supplementary material associated with this article can be found, in the online version, at 10.1016/j.ipm.2018.03.006

References

von Ahn, L. (2005). Human computation. Carnegie Mellon University Ph.D. thesis.von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8), 57. http://dx.doi.org/10.1145/1378704.1378719.Aigrain, J., Dapogny, A., Bailly, K., Dubuisson, S., Detyniecki, M., & Chetouani, M. (2016). On leveraging crowdsourced data for automatic perceived stress detection.

Proceedings of the 18th ACM international conference on multimodal interactionICMI 2016New York, NY, USA: ACM113–120. http://dx.doi.org/10.1145/2993148.2993200.

Alonso, O., & Mizzaro, S. (2012). Using crowdsourcing for trec relevance assessment. Information Processing & Management, 48(6), 1053–1066. http://dx.doi.org/10.1016/j.ipm.2012.01.004.

Altmeyer, M., Lessel, P., & Krüger, A. (2016). Expense control: A gamified, semi-automated, crowd-based approach for receipt capturing. Proceedings of the 21st internationalconference on intelligent user interfaces – IUI ’16IUI ’1607-10-Marc. Proceedings of the 21st international conference on intelligent user interfaces – IUI ’16 New York, NewYork, USA: ACM Press31–42. http://dx.doi.org/10.1145/2856767.2856790.

Ashikawa, M., Kawamura, T., & Ohsuga, A. (2015). Deployment of private crowdsourcing system with quality control methods. 2015 IEEE/WIC/ACM international conferenceon web intelligence and intelligent agent technology (WI-IAT)1. 2015 IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT) IEEE9–16. http://dx.doi.org/10.1109/WI-IAT.2015.81.

Ashley, H., Corbett, J., Jones, D., Garside, B., & Rambaldi, G. (2009). Change at hand: Web 2.0 for development. Participatory Learning and Action, 59, 8–20.Banks, S., Rafter, R., & Smyth, B. (2015). The recommendation game: Using a game-with-a-purpose to generate recommendation data. Proceedings of the 9th ACM conference

on recommender systemsRecSys ’15New York, NY, USA: ACM305–308. http://dx.doi.org/10.1145/2792838.2799675.Bayas, J. C. L., See, L., Fritz, S., Sturn, T., Perger, C., Dürauer, M., et al. (2016). Crowdsourcing in-situ data on land cover and land use using gamification and mobile

technology. Remote Sensing, 8(11), http://dx.doi.org/10.3390/rs8110905.Brabham, D. C. (2012). Crowdsourcing: A model for leveraging online communities. In A. Delwiche, & J. J. Henderson (Eds.). The participatory cultures handbook (pp.

120–129). (Routledge ed.). Taylor & Francis.Brambilla, M., Ceri, S., Mauri, A., & Volonterio, R. (2014). Community-based crowdsourcing. Proceedings of the 23rd international conference on world wide web – www ’14

companionWWW ’14 CompanionNew York, New York, USA: ACM Press891–896. http://dx.doi.org/10.1145/2567948.2578835.Bénabou, R., & Tirole, J. (2003). Intrinsic and extrinsic motivation. The Review of Economic Studies, 70(3), 489–520.Can, G., Odobez, J.-M., & Gatica-Perez, D. (2014). Is that a jaguar?: Segmenting ancient maya glyphs via crowdsourcing. Proceedings of the 2014 international ACM workshop

on crowdsourcing for multimediaCrowdMM ’14New York, NY, USA: ACM37–40. http://dx.doi.org/10.1145/2660114.2660117.Cao, C. C., Chen, L., & Jagadish, H. V. (2014). From labor to trader: Opinion elicitation via online crowds as a market. Proceedings of the 20th ACM SIGKDD international

conference on knowledge discovery and data mining – KDD ’14KDD ’14New York, New York, USA: ACM Press1067–1076. http://dx.doi.org/10.1145/2623330.2623717.

Castano, S., Ferrara, A., Genta, L., & Montanelli, S. (2016). Combining crowd consensus and user trustworthiness for managing collective tasks. Future GenerationComputer Systems, 54, 378–388. http://dx.doi.org/10.1016/j.future.2015.04.014.

Chandler, D., & Kapelner, A. (2013). Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization, 90,123–133. http://dx.doi.org/10.1016/j.jebo.2013.03.003.

Chowdhury, S. A., Ghosh, A., Stepanov, E., Bayer, A. O., Riccardi, G., & Klasinas, I. (2014). Cross-language transfer of semantic annotation via targeted crowdsourcing.In E. S. Cheng, H. Li, H. Meng, B. Ma, & L. Xie (Eds.). Proceedings of the annual conference of the international speech communication association, interspeech, September2015 (pp. 2108–2112). Dept. of Information Engineering and Computer Science, Univ. of Trento, Trento, Italy: International Speech and CommunicationAssociation.

Della Mea, V., Maddalena, E., Mizzaro, S., Machin, P., & Beltrami, C. A. (2014). Preliminary results from a crowdsourcing experiment in immunohistochemistry.Diagnostic Pathology, 9(1), 1–6. http://dx.doi.org/10.1186/1746-1596-9-S1-S6.

Di Salvo, R., Spampinato, C., & Giordano, D. (2016). Generating reliable video annotations by exploiting the crowd. 2016 IEEE winter conference on applications of computervision (WACV). IEEE1–8. http://dx.doi.org/10.1109/WACV.2016.7477718.

Dietl, W., Dietzel, S., Ernst, M. D., Mote, N., Walker, B., Cooper, S., et al. (2012). Verification games: Making verification fun. Proceedings of the 14th workshop on formaltechniques for java-like programs – FTfJP ’12FTfJP ’12New York, NY, USA: ACM Press42–49. http://dx.doi.org/10.1145/2318202.2318210.

Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86. http://dx.doi.org/10.1145/1924421.1924442.

Dontcheva, M., Morris, R. R., Brandt, J. R., & Gerber, E. M. (2014). Combining crowdsourcing and learning to improve engagement and performance. Proceedings of the 32ndannual ACM conference on human factors in computing systemsCHI ’14New York, NY, USA: ACM3379–3388. http://dx.doi.org/10.1145/2556288.2557217.

Dumitrache, A., Aroyo, L., Welty, C., Sips, R.-J., & Levas, A. (2013). ”dr. detective”: Combining gamification techniques and crowdsourcing to create a gold standard inmedical text. In M. Acosta, L. Aroyo, A. Bernstein, J. Lehmann, N. Noy, & E. Simperl (Vol. Eds.), 1st international workshop on crowdsourcing the semantic web,CrowdSem 2013. 1030. 1st international workshop on crowdsourcing the semantic web, CrowdSem 2013 (pp. 16–31). VU University Amsterdam, Netherlands:CEUR-WS.

Dwarakanath, A., Chintala, U., Shrikanth, N., Virdi, G., Kass, A., Chandran, A., et al. (2015). Crowd build: A methodology for enterprise software development usingcrowdsourcing. 2015 IEEE/ACM 2nd international workshop on CrowdSourcing in software engineering (CSI-SE)CSI-SE ’15Piscataway, NJ, USA: IEEE8–14. http://dx.

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

504

doi.org/10.1109/CSI-SE.2015.9.Enochson, K., & Culbertson, J. (2015). Collecting psycholinguistic response time data using amazon mechanical turk. PLOS ONE, 10(3), 1–17. http://dx.doi.org/10.

1371/journal.pone.0116946.Estellés-Arolas, E., & González-Ladrón-de Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38(2), 189–200. http://

dx.doi.org/10.1177/0165551512437638.Estellés-Arolas, E., Navarro-Giner, R., & González-Ladrón-de Guevara, F. (2015). Crowdsourcing fundamentals: Definition and typology. In F. J. Garrigos-Simon, I. Gil-

Pechuán, & S. Estelles-Miguel (Eds.). [Advances in Crowdsourcing]. Cham: Springer International Publishing. http://dx.doi.org/10.1007/978-3-319-18341-1_3.Estes, L. D., McRitchie, D., Choi, J., Debats, S., Evans, T., Guthe, W., et al. (2016). A platform for crowdsourcing the creation of representative, accurate landcover

maps. Environmental Modelling & Software, 80, 41–53. http://dx.doi.org/10.1016/j.envsoft.2016.01.011.Fava, D., Shapiro, D., Osborn, J., Schäef, M., & Whitehead Jr., E. J. (2016). Crowdsourcing program preconditions via a classification game. Proceedings of the 38th

international conference on software engineering – ICSE ’16ICSE ’16New York, New York, USA: ACM Press1086–1096. http://dx.doi.org/10.1145/2884781.2884865.Feng, S., Woo, M.-J., Kim, H., Kim, E., Ki, S., & Shao, L. (2016). A game-based crowdsourcing platform for rapidly training middle and high school students to perform

biomedical image analysis. In D. Levitz, A. Ozcan, & D. Erickson (Vol. Eds.), Progress in biomedical optics and imaging – Proceedings of SPIE. 9699. Progress inbiomedical optics and imaging – Proceedings of SPIE (pp. 1–9). Electrical Engineering Department, David Geffen School of Medicine, University of California, LosAngeles, CA, United States: SPIE. http://dx.doi.org/10.1117/12.2212310.

Figuerola Salas, Ó., Adzic, V., Kalva, H., & Shah, A. (2013). Assessing internet video quality using crowdsourcing. Proceedings of the 2nd ACM international workshop oncrowdsourcing for multimedia – CrowdMM ’13CrowdMM ’13New York, New York, USA: ACM Press23–28. http://dx.doi.org/10.1145/2506364.2506366.

Foncubierta Rodríguez, A., & Müller, H. (2012). Ground truth generation in medical imaging a. Proceedings of the ACM multimedia 2012 workshop on crowdsourcing formultimedia – CrowdMM ’12CrowdMM ’12New York, New York, USA: ACM Press9–14. http://dx.doi.org/10.1145/2390803.2390808.

Geiger, D., Fielt, E., Rosemann, M., & Schader, M. (2012). Crowdsourcing information systems-definition, typology, and design. Proceedings of the 33rd internationalconference on information systems. 2012. Association for information systems AIS electronic library (AISEL).1–11.

Geiger, D., Seedorf, S., Schulze, T., Nickerson, R. C., & Schader, M. (2011). Managing the crowd: Towards a taxonomy of crowdsourcing processes. Proceedings of the 17thAmericas conference on information systems (AMCIS 2011). Atlanta, Ga.: AISeL Online: http://aisel.aisnet.org/cgi/viewcontent.cgi?article (p. Paper 430). URL:http://ub-madoc.bib.uni-mannheim.de/29003/ online Resource

Gupta, A., Thies, W., Cutrell, E., & Balakrishnan, R. (2012). mclerk: Enabling mobile crowdsourcing in developing regions. Proceedings of the 2012 ACM annual conference onhuman factors in computing systems – CHI ’12CHI ’12New York, New York, USA: ACM Press1843–1852. http://dx.doi.org/10.1145/2207676.2208320.

Hester, V., Shaw, A., & Biewald, L. (2010). Scalable crisis relief: Crowdsourced sms translation and categorization with mission 4636. Proceedings of the first ACM symposiumon computing for development – ACM DEV ’10ACM DEV ’10New York, New York, USA: ACM Press1–7. http://dx.doi.org/10.1145/1926180.1926199.

Hetmank, L. (2013). Components and functions of crowdsourcing systems—a systematic literature review. 11th international conference on wirtschaftsinformatik, March55–69.http://dx.doi.org/10.13140/2.1.3836.4166.

Hillen, F., & Höfle, B. (2015). Geo-recaptcha: Crowdsourcing large amounts of geographic information from earth observation data. International Journal of AppliedEarth Observation and Geoinformation, 40, 29–38. http://dx.doi.org/10.1016/j.jag.2015.03.012.

Hosseini, M., Phalp, K., Taylor, J., & Ali, R. (2014). The four pillars of crowdsourcing: A reference model. 2014 IEEE eighth international conference on research challenges ininformation science (RCIS). IEEE1–12. http://dx.doi.org/10.1109/RCIS.2014.6861072.

Hosseini, M., Shahri, A., Phalp, K., Taylor, J., & Ali, R. (2015). Crowdsourcing: A taxonomy and systematic mapping study. Computer Science Review, 17, 43–69. http://dx.doi.org/10.1016/j.cosrev.2015.05.001.

Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14 Available at https://www.wired.com/2006/06/crowds Accessed in: Sep. 15, 2017Howe, J. (2008). CROWDSOURCING: Why the power of the crowd is driving the future of business (1st). Crown Busines.Huynh, A., Eguchi, M., Lin, A. Y.-M., & Eguchi, R. (2014). Limitations of crowdsourcing using the ems-98 scale in remote disaster sensing. 2014 IEEE aerospace conference.

IEEE1–7. http://dx.doi.org/10.1109/AERO.2014.6836457.Irshad, H., Montaser-Kouhsari, L., Waltz, G., Bucur, O., Nowak, J. A., Dong, F., et al. (2015). Crowdsourcing image annotation for nucleus detection and segmentation in

computational pathology: Evaluating experts, automated methods, and the crowd. 20th Pacific symposium on biocomputing, PSB 2015. Beth Israel Deaconess MedicalCenter, Harvard Medical School, Boston, United States: Stanford University294–305.

Irshad, H., Oh, E.-Y., Schmolze, D., Quintana, L. M., Collins, L., Tamimi, R. M., et al. (2017). Crowdsourcing scoring of immunohistochemistry images: Evaluatingperformance of the crowd and an automated computational method. Scientific Reports, 7, 43286. http://dx.doi.org/10.1038/srep43286.

Kermanidis, K. L., Maragoudakis, M., & Vosinakis, S. (2015). House of ads: a multiplayer action game for annotating ad video content. Proceedings of the 16th internationalconference on engineering applications of neural networks (INNS) – EANN ’15EANN ’15New York, New York, USA: ACM Press1–8. http://dx.doi.org/10.1145/2797143.2797174.

Kim, J., Kim, S., Yang, J., Ryu, J.-h., & Wohn, K. (2014). Facecaptcha: A captcha that identifies the gender of face images unrecognized by existing gender classifiers.Multimedia Tools and Applications, 72(2), 1215–1237. http://dx.doi.org/10.1007/s11042-013-1422-z.

Kim, S.-H., Li, S., Chul Kwon, B., & Yi, J. S. (2011). Investigating the efficacy of crowdsourcing on evaluating visual decision supporting system. Proceedings of the humanfactors and ergonomics society 55th annual meeting55. Proceedings of the human factors and ergonomics society 55th annual meeting School of Industrial Engineering,Purdue University, United States1090–1094. http://dx.doi.org/10.1177/1071181311551228.

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., et al. (2013). The future of crowd work. Proceedings of the 2013 conference on computersupported cooperative workCSCW ’13New York, NY, USA: ACM1301–1318. http://dx.doi.org/10.1145/2441776.2441923.

Kulkarni, A., Can, M., & Hartmann, B. (2012). Collaboratively crowdsourcing workflows with turkomatic. Proceedings of the ACM 2012 conference on computer supportedcooperative work – CSCW ’12. New York, New York, USA: ACM Press1003. http://dx.doi.org/10.1145/2145204.2145354.

Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., et al. (2012). Real-time captioning by groups of non-experts. Proceedings of the 25thannual acm symposium on user interface software and technology – UIST ’12UIST ’12New York, New York, USA: ACM Press23–33. http://dx.doi.org/10.1145/2380116.2380122.

Lasecki, W. S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J. F., & Bigham, J. P. (2013). Chorus: A crowd-powered conversational assistant walter. Proceedings of the 26thannual acm symposium on user interface software and technology – UIST ’13UIST ’13New York, New York, USA: ACM Press151–162. http://dx.doi.org/10.1145/2501988.2502057.

Law, E., & Ahn, L. V. (2011). Human computation. 5Morgan & Claypool Publishershttp://dx.doi.org/10.2200/S00371ED1V01Y201107AIM013.Leano, R., Wang, Z., & Sarma, A. (2016). Labeling relevant skills in tasks: Can the crowd help? Proceedings of IEEE symposium on visual languages and human-centric

computing, VL/HCC2016-Novem. Proceedings of IEEE symposium on visual languages and human-centric computing, VL/HCC 185–189 Cambridge, United KingdomLeicht, N., Knop, N., Blohm, I., Müller-Bloch, C., & Leimeister, J. M. (2016). When is crowdsourcing advantageous? the case of crowdsourced software testing. 24th European

conference on information systems, ECIS 2016. Association for Information Systems.Loni, B., Cheung, L. Y., Riegler, M., Bozzon, A., Gottlieb, L., & Larson, M. (2014). Fashion 10000: An enriched social image dataset for fashion and clothing. Proceedings of

the 5th ACM multimedia systems conference on – MMSYS ’14MMSys ’14New York, New York, USA: ACM Press41–46. http://dx.doi.org/10.1145/2557642.2563675.Maier-Hein, L., Mersmann, S., Kondermann, D., Bodenstedt, S., Sanchez, A., Stock, C., et al. (2014). Can masses of non-experts train highly accurate image classifiers? a

crowdsourcing approach to instrument segmentation in laparoscopic images. Medical image computing and computer-assisted intervention : MICCAI ... International con-ference on medical image computing and computer-assisted intervention17. Medical image computing and computer-assisted intervention : MICCAI ... International con-ference on medical image computing and computer-assisted intervention 438–445. http://dx.doi.org/10.1007/978-3-319-10470-6_55 PART 2

Malone, T. W., Laubacher, R., & Dellarocas, C. (2010). The collective intelligence genome. IEEE Engineering Management Review, 38(3), 38–52. http://dx.doi.org/10.1109/EMR.2010.5559142.

Masiar, A., & Simko, J. (2015). Short video metadata acquisition game. 2015 10th international workshop on semantic and social media adaptation and personalization(SMAP). IEEE61–65. http://dx.doi.org/10.1109/SMAP.2015.7370092.

McAllister Byun, T., Halpin, P. F., & Szeredi, D. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders, 53,

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

505

70–83. http://dx.doi.org/10.1016/j.jcomdis.2014.11.003.McNaney, R., Othman, M., Richardson, D., Dunphy, P., Amaral, T., Miller, N., et al. (2016). Speeching: Mobile crowdsourced speech assessment to support self-monitoring

and management for people with parkinson’s. Proceedings of the 2016 CHI conference on human factors in computing systems – CHI ’16CHI ’16New York, New York, USA:ACM Press4464–4476. http://dx.doi.org/10.1145/2858036.2858321.

Mellebeek, B., Benavent, F., Grivolla, J., Codina, J., Costa-jussà, M. R., & Banchs, R. (2010). Opinion mining of spanish customer comments with non-expert annotations onmechanical turk. Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s mechanical turkCSLDAMT ’10Stroudsburg, PA,USA: Association for Computational Linguistics114–121.

Nguyen-Dinh, L.-V., Waldburger, C., Roggen, D., & Tröster, G. (2013). Tagging human activities in video by crowdsourcing. Proceedings of the 3rd ACM conference oninternational conference on multimedia retrieval – ICMR ’13ICMR ’13New York, New York, USA: ACM Press263–270. http://dx.doi.org/10.1145/2461466.2461508.

Noronha, J., Hysen, E., Zhang, H., & Gajos, K. Z. (2011). Platemate: Crowdsourcing nutrition analysis from food photographs. Proceedings of the 24th annual ACM symposiumon user interface software and technology – UIST ’11UIST ’11New York, New York, USA: ACM Press1–11. http://dx.doi.org/10.1145/2047196.2047198.

Oleson, D., Sorokin, A., Laughlin, G., Hester, V., Le, J., & Biewald, L. (2011). Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Proceedings ofthe 11th AAAI conference on human computationAAAIWS’11-11AAAI Press43–48.

Pastore, F., Mariani, L., & Fraser, G. (2013). Crowdoracles: Can the crowd solve the oracle problem? 2013 IEEE sixth international conference on software testing, verificationand validation. IEEE342–351. http://dx.doi.org/10.1109/ICST.2013.13.

Ramchurn, S. D., Huynh, T. D., Venanzi, M., & Shi, B. (2013). Collabmap: Crowdsourcing maps for emergency planning. Proceedings of the 5th annual ACM web scienceconference on – WebSci ’13volume of WebSci ’13New York, New York, USA: ACM Press326–335. http://dx.doi.org/10.1145/2464464.2464508.

Redi, J., Hoßfeld, T., Korshunov, P., Mazza, F., Povoa, I., & Keimel, C. (2013). Crowdsourcing-based multimedia subjective evaluations: A case study on image recognizabilityand aesthetic appeal. Proceedings of the 2nd ACM international workshop on crowdsourcing for multimedia – CrowdMM ’13CrowdMM ’13New York, New York, USA:ACM Press29–34. http://dx.doi.org/10.1145/2506364.2506368.

Retelny, D., Bernstein, M. S., & Valentine, M. A. (2017). No workflow can ever be enough: How crowdsourcing workflows constrain complex work. Proceedings of the ACM onhuman-computer interaction. New York, NY, USA: ACM23. http://dx.doi.org/10.1145/3134724 Vol. 1

Roemmele, M., Archer-McClellan, H., & Gordon, A. S. (2014). Triangle charades: A data-collection game for recognizing actions in motion trajectories. Proceedings of the 19thinternational conference on intelligent user interfaces – IUI ’14IUI ’14New York, New York, USA: ACM Press209–214. http://dx.doi.org/10.1145/2557500.2557510.

Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. Computer supported collaborative learning. Berlin, Heidelberg:Springer Berlin Heidelberg69–97. http://dx.doi.org/10.1007/978-3-642-85098-1_5.

Ross, J., Zaldivar, A., Irani, L., & Tomlinson, B. (2010). Who are the turkers? Worker demographics in amazon mechanical turk. Chi Ea 20102863–2872. http://dx.doi.org/10.1145/1753846.1753873.

Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25(1), 54–67. http://dx.doi.org/10.1006/ceps.1999.1020.

Sabou, M., Scharl, A., & Föls, M. (2013). Crowdsourced knowledge acquisition: Towards hybrid-genre workflows. International Journal on Semantic Web and InformationSystems, 9(3), 14–41. http://dx.doi.org/10.4018/ijswis.2013070102.

Soleymani, M. (2015). The quest for visual interest. Proceedings of the 23rd ACM international conference on multimedia – MM ’15MM ’15New York, New York, USA: ACMPress919–922. http://dx.doi.org/10.1145/2733373.2806364.

Staffelbach, M., Sempolinski, P., Kijewski-Correa, T., Thain, D., Wei, D., Kareem, A., et al. (2015). Lessons learned from crowdsourcing complex engineering tasks.PLOS ONE, 10(9), e0134978. http://dx.doi.org/10.1371/journal.pone.0134978.

Sulser, F., Giangreco, I., & Schuldt, H. (2014). Crowd-based semantic event detection and video annotation for sports videos. Proceedings of the 2014 international ACMworkshop on crowdsourcing for multimedia – CrowdMM ’14CrowdMM ’14New York, New York, USA: ACM Press63–68. http://dx.doi.org/10.1145/2660114.2660119.

Taborsky, E., Allen, K., Blanton, A., Jain, A. K., & Klare, B. F. (2015). Annotating unconstrained face imagery: A scalable approach. 2015 International conference onbiometrics (ICB). IEEE264–271. http://dx.doi.org/10.1109/ICB.2015.7139094.

Tavares, G., Mourão, A., & Magalhaes, J. (2013). Crowdsourcing for affective-interaction in computer games. Proceedings of the 2nd ACM international workshop oncrowdsourcing for multimedia – CrowdMM ’13CrowdMM ’13New York, New York, USA: ACM Press7–12. http://dx.doi.org/10.1145/2506364.2506369.

To, H., Geraldes, R., Shahabi, C., Kim, S. H., & Prendinger, H. (2016). An empirical study ofworkers’ behavior in spatial crowdsourcing. Proceedings of the third internationalACM sigmod workshop on managing and mining enriched geo-spatial data – GeoRich ’16GeoRich ’16New York, New York, USA: ACM Press1–6. http://dx.doi.org/10.1145/2948649.2948657.

Tuite, K., Snavely, N., Hsiao, D.-Y., Tabing, N., & Popović, Z. (2011). Photocity: Training experts at large-scale image acquisition through a competitive game. Proceedings ofthe 2011 annual conference on human factors in computing systems – CHI ’11CHI ’11New York, New York, USA: ACM Press1383–1392. http://dx.doi.org/10.1145/1978942.1979146.

Vernez, S. L., Huynh, V., Osann, K., Okhunov, Z., Landman, J., & Clayman, R. V. (2017). C-Sats: Assessing surgical skills among urology residency applicants. Journal ofEndourology, 31(S1), S–95–S–100. http://dx.doi.org/10.1089/end.2016.0569.

Vukovic, M., Salapura, V., & Rajagopal, S. (2013). Crowd-enabled technical writing services. 2013 10th IEEE international conference on services computing. IEEE635–642.http://dx.doi.org/10.1109/SCC.2013.84.

Weidema, E. R., López, C., Nayebaziz, S., Spanghero, F., & van der Hoek, A. (2016). Toward microtask crowdsourcing software design work. Proceedings of the 3rdinternational workshop on crowdsourcing in software engineering – CSI-SE ’16CSI-SE ’16New York, New York, USA: ACM41–44. http://dx.doi.org/10.1145/2897659.2897664.

Yang, Z., Li, B., Zhu, Y., King, I., Levow, G., & Meng, H. (2010). Collection of user judgments on spoken dialog system with crowdsourcing. 2010 IEEE spoken languagetechnology workshop, Berkeley, CA, United States. IEEE277–282. http://dx.doi.org/10.1109/SLT.2010.5700864.

Yin, M., Gray, M. L., Suri, S., & Vaughan, J. W. (2016). The communication network within the crowd. Proceedings of the 25th international conference on world wide web –WWW ’16WWW ’16New York, New York, USA: ACM Press1293–1303. http://dx.doi.org/10.1145/2872427.2883036.

Zhai, H., Lingren, T., Deleger, L., Li, Q., Kaiser, M., Stoutenborough, L., et al. (2013). Web 2.0-based crowdsourcing for high-quality gold standard development inclinical natural language processing. Journal of Medical Internet Research, 15(4), 1–17. http://dx.doi.org/10.2196/jmir.2426.

Zhai, Z., Kijewski-Correa, T., Hachen, D., & Madey, G. (2012). Haiti earthquake photo tagging: Lessons on crowdsourcing in-depth image classification. Seventh internationalconference on digital information management (ICDIM 2012). IEEE357–364. http://dx.doi.org/10.1109/ICDIM.2012.6360130.

Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., & Horvitz, E. (2012). Human computation tasks with global constraints. Proceedings of the 2012 ACM annual conferenceon human factors in computing systems – CHI ’12. New York, New York, USA: ACM Press217. http://dx.doi.org/10.1145/2207676.2207708.

Zhao, Y., & Zhu, Q. (2014). Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers, 16(3), 417–434. http://dx.doi.org/10.1007/s10796-012-9350-4.

F.R. Assis Neto, C.A.S. Santos Information Processing and Management 54 (2018) 490–506

506