Printing:

This poster is 48” wide by 36” high. It’s designed to be printed on a large-format printer.

Customizing the Content:

The placeholders in this poster are formatted for you. Type in the placeholders to add text, or click an icon to add a table, chart, SmartArt graphic, picture or multimedia file.

To add or remove bullet points from text, just click the Bullets button on the Home tab.

If you need more placeholders for titles, content or body text, just make a copy of what you need and drag it into place. PowerPoint’s Smart Guides will help you align it with everything else.

Want to use your own pictures instead of ours? No problem! Just right-click a picture and choose Change Picture. Maintain the proportion of pictures as you resize by dragging a corner.

Massive Online Experiment NeedsN1: Implement many experimental designs in an online format

N2: Recruit and reward participants without providing cash payment or course credit

N3: Ensure that the study is available all the time (no downtime)

N4: Store the data being collected while the study is active

N5: (bonus) Implement sophisticated stimulus selection logic!

AN OPEN-SOURCE ENGINE FOR SOCIAL SCIENCE AT SCALE

Timeline and Future Directions

Mariela V. Jennings1, Joshua K. Hartshorne1

1Department of Psychology, Boston College

Interested in large scale data collection? Pushkin is for conducting massive online experiments!

A collection of tools and technologies that can be assembled into a versatile platform for running successful massive online experiments (see gameswithwords.org). Pushkin is designed to provide the majority of the functionality needed to conduct successful massive online experiments while offering flexibility and modularity that allows for easy modifications and customization.

What is Pushkin?

Experiments:

Webpages:

Database Workers: Write to and read from the database; handle seeding the database with an initial list of stimuli for each study

Task Workers: Select and order stimuli before presenting them to the participant; analyze data in real time; use participant responses to provide feedback; can be used to implement OED

API: A routing layer that points to other services Message Queue: Enables communication between services in different programming languages

Cron Worker: Runs (Python or JavaScript) scripts periodically; can be used to analyze data and send reports or remove user IDs for inactive users

Load Balancer: Distributes website traffic to the different copies of each Pushkin service created via scaling.

1. Services: Scaling by making additional copies of bottleneck services on the same host

2. Hosts: Scaling by making additional copies of all Pushkin backend services on a new host

Acknowledgements

Encouraging discussions in crowdsourcing projects: forum

Longitudinal studies/increasing participant return rates: social media authentication, user dashboard

Standard vs Pushkin Experiment

N1

N2

N2

N1, N2

N4

N3

Extra feature!

N5

(Auto)scalingWhy is scaling necessary?

What kinds of scaling are there?

Where Do We Get Subjects From?

A Pushkin experiment allows for flexible, real-time stimulus selection and contingent designs

Under development: • Web-based eye tracking (WebGazer.js) • Percentile score feedback template • Additional feedback templates

Challenges currently being addressed: • Faster and simpler deployment of a Pushkin website • Reusable templates for scientists with little

programming knowledge • Procedures for keeping Pushkin dependencies up-to-

date • More extensive documentation

Rewards for participation: personalized feedback & social media sharing; gamification (leaderboards, badges)

Database: Contains stimuli and responses Data Log: Keeps a permanent record of all queries; implements version control N4