Response option order effects.

Scale lengths and horizontal or vertical layout.

Johan MartinssonUniversity of Gothenburgwww.lore.gu.se

Overview

• 2 related but separate studies

• The general focus on effects of different ways to arrange response options on a screen (eg. ”primacy effects”, tendency to pick the first response option)

• Three factors to consider:

• scale length

• layout direction (horizontal or vertical)

• response device (PC, tablet, smartphone)

www.mod.gu.se

Why do we care?

• You want a neutral measurement instrument

• An arbitrary design choice on your part should not influence results

• An arbitrary choice of device of respondents should not influence results

• => robust estimates

www.mod.gu.se

Some background

• More and more respondents use smartphone to answer surveys (though clearly a minority)

• Smartphone compatibility

no classic gridsshorter scalesvertical layout

• Screen sizes differ

www.mod.gu.se

Some background

• More automatic adjustment to device by survey softwares (responsive design)

Different respondents get different design

Robustness of measurement instruments more important to examine

www.mod.gu.sewww.mod.gu.se

Examples

Reversing response option orders

Horizontal or vertical layout

Example of automatic mobile adjustment by survey software

www.mod.gu.se

Study 1

• Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014

• Field work: July to September 2014• Total n approximately 20,000

• Randomized experimental design 2x2x2: • ”Right direction” or ”wrong direction” first• 7 or 3 scale points• Horizontal or vertical layout

www.mod.gu.se

Study 1

• We had a few hypotheses... : primacy effects exist stronger effects with vertical layouts than horizontalstronger effects for smartphone users(especially with horizontal layouts)

www.mod.gu.se

Results

All respondents7-point scale with horizontal layout

PlacementFirst Last diff

Right direction 11 10 1Wrong directionn 2561 2538

www.mod.gu.se

Results

All respondents7-point scale with horizontal layout

PlacementFirst Last diff

Right direction 11 10 1Wrong direction 19 21 -2n 2561 2538

www.mod.gu.se

all respondents

7-point scale with vertical layout

Placement

First Last diff

Right direction 15

Wrong direction 26

n 2457 2455

www.mod.gu.se

all respondents

7-point scale with vertical layout

Placement

First Last diff

Right direction 15 11 4

Wrong direction 26 21 5

n 2457 2455

www.mod.gu.se

all respondents

3-point scale with horizontal layout

Placement

First Last diff

Right direction 31 28 3Wrong direction 49 45 4

n 2421 2492

www.mod.gu.se

all respondents

3-point scale with vertical layout

Placement

First Last diff

Right direction 32 28 4Wrong direction 48 47 1

n 2566 2550

www.mod.gu.se

Kind of boring results, no?

• Luckily, we discovered something more interesting by mistake!

www.mod.gu.se

7-point scale with horizontal layoutPlacementFirst Last

Right direction 11 10Wrong direction 19 21n 2561 2538

7-point scale with vertical layout

Placement

First Last

Right direction 15 11

Wrong direction 26 21

n 2457 2455

www.mod.gu.se

3-point scale with vertical layout

Placement

First Last

Right direction 32 28

Wrong direction 48 47

n 2566 2550

3-point scale with horizontal layoutPlacementFirst Last

Right direction 31 28

Wrong direction 49 45n 2421 2492

www.mod.gu.se

Study 2

• Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014

• Field work: July to September 2014• Total n approximately 34,000

• Randomized experimental design 5x2(x2): • Horizontal or vertical layout of the questions• 10, 7, 5, 3, or 2 scale points (response options)

www.mod.gu.sewww.mod.gu.se

Percent "Wrong direction" among everyoneHorizontal Vertical Difference

10-point scale7-point scale5-point scale3-point scale2-point scale

Results

www.mod.gu.sewww.mod.gu.se

Percent "Wrong direction" among everyoneHorizontal Vertical Difference

10-point scale 16 21 -57-point scale 20 27 -75-point scale 24 31 -73-point scale2-point scale

Results

www.mod.gu.sewww.mod.gu.se

Percent "Wrong direction" among everyoneHorizontal Vertical Difference

10-point scale 16 21 -57-point scale 20 27 -75-point scale 24 31 -73-point scale 49 49 02-point scale 58 59 -1

Results

www.mod.gu.se

Percent "Wrong direction" among PC users

HorizontalVertical Difference10-point scale 18 20 -27-point scale 21 27 -65-point scale 27 31 -43-point scale 50 50 02-point scale 60 60 0

www.mod.gu.se

Percent "Wrong direction" among smartphone users

Horizontal Vertical Difference10-point scale 14 25 -117-point scale 19 29 -105-point scale 21 33 -123-point scale2-point scale

www.mod.gu.se

Percent "Wrong direction" among smartphone users

Horizontal Vertical Difference10-point scale 14 25 -117-point scale 19 29 -105-point scale 21 33 -123-point scale 50 51 -12-point scale 59 61 -2

www.mod.gu.se

Difference depending on device

All PC Smartphone10-point scale7-point scale5-point scale3-point scale2-point scale

www.mod.gu.se

Difference depending on device

All PC Smartphone10-point scale -5 -2 -117-point scale -7 -6 -105-point scale -7 -4 -123-point scale 0 0 -12-point scale -1 0 -2

www.mod.gu.se

Conclusions and recommendations

• We observe weak but fairly robust primacy effects (as usual)

• These do not seem to depend on device or scale length much (more surprisingly)

www.mod.gu.se

First, some useless advice...

• Ban smartphones in your surveys!

• Only use horizontal scales (not even sure it is better)

www.mod.gu.se

Some less useless advice

• Use really short scales

• Randomize when you can (nominal scales)

• Split half reversed order?

• Use long questions wordings and make sure respondents are aware of all available options before allowing them to answer (but increases burden and time)

www.mod.gu.se

Read more at

www.lore.gu.se

Laboratory of Opinion Research (LORE)