What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human

Computation in a Spatial Location Identification Task

What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human

Computation in a Spatial Location Identification Task

Huaming Rao, Shih-Wen Huang, Wai-Tat FuCascade Lab, Department of Computer Science

Huaming Rao, Shih-Wen Huang, Wai-Tat FuCascade Lab, Department of Computer Science

Agenda

✦ Motivation

✦ Experiment Design and Expectation

✦ Result

✦ Conclusion and Future Work

Motivation

a Spatial Location Identification Task (SpLIT)

Motivation

Motivation✦ Human computation

✦ Coordination tasks

✦ Two control variables

✦ Difficulty levels

Experiment Design

Experiment Design

Different difficulty levels of the pictures

✦ Two control variables

✦ Difficulty levels

✦ Two reward schemes: ground truth and majority vote.

Experiment Design

Majority Vote Scheme

Ground Truth Scheme

Experiment Design

Experiment Design

Experimental interface

Expectation✦ A spam filtering effect

✦ A reflection effect

Results✦ Recruited 103 workers:

✦ 50 for ground truth

✦ 53 for majority vote

✦ Valid workers

✦ 41 for ground truth

✦ 43 for majority vote

Results

Accuracies of SpLIT in each of the tasks

Results

Cumulative histogram of the accuracies of individual workers

Results

Average within-cluster distances

Pic Num. 1 2 3 4 5

Ground Truth

46.94

30.76

39.53

19.33

39.24

Majority Vote

44.03

22.51

39.71

13.13

28.44

Significantly smaller in majority vote

Results

Percentage of points in each cluster

Results

Ground Truth Majority Vote

The clusters in the scheme of majority vote had higher percentages and smaller averages distances to the center

Results✦ The majority vote scheme:

✦ Higher accuracies of each task

✦ Larger percentage of workers with high accuracies

✦ Smaller average distances of each cluster

✦ Clusters with highest percentage more likely the correct ones

Conclusion

A notational figure showing the distributions of answers

Conclusion✦ The majority vote scheme can lead to

higher precision and reliability when identifying spatial locations

✦ Small dataset but big effect and significant differences

✦ Human computation can achieve at some accuracies while performing SpLIT

✦ Spammer rates almost the same

Future Work

✦ Experiment to test what leads to current result

✦ Incorporates humans into complex graphic or spatial computations applications

Experiment DesignYou will be paid based on your

performance, which is measured by whether the pin location that you placed is close enough to the actual location

where the picture was taken. You also need to provide an

acceptable description of how you figure out the

location.

Experiment DesignYou will be paid based on your

performance, which is measured by whether the pin location that you placed is close enough to the majority of the

locations by other turkers. You also

need to provide an acceptable

description of how you figure out the

location.

Discussion I

One Exception

Ground Truth

Majority Vote

Discussion I

One Exception

Discussion I

✦ “I could see an exit symbol at a corner in the picture with that I have assumed that it must be the place.”

✦ “I did it based on the long corridor and what looks to be an exit at the end.”

Discussion I

Discussion I

✦ When some workers believe that a wrong answer or wrong cue could be more salient than the correct answer to most other workers, they may choose the wrong one instead of the correct one.