Enhancements to the National Rail Passenger Survey€¦ · paid envelope Passenger is offered...

Enhancements to

the National Rail

Passenger Survey

Findings from pilot work

during Spring 2016

(Cut down for Experts Group)

June 2016

2

Contents

Background 3

Overview of findings 11

Impact of enhancements on response rate 13

Impact of enhancements on who takes part 19

Impact of enhancements on satisfaction results 21

Impact of enhancements on „missing‟ questions 25

New questions: trust and how people feel 30

Evaluating respondent experience of the survey further 39

Summary 58

Background to this research

4

Background & objectives

Following industry

consultation, Transport

Focus has used the Spring

2016 wave of NRPS to pilot

potential changes to the

survey’s method:

• Shortened version of the

questionnaire, with some

enhancements to layout /

aesthetics, and a small

number of potential new

questions

• Two data collection options:

passengers have the choice

to take part on paper or

online

This report covers the findings from this pilot, based on two

strands of research:

Quantitative evaluation of the changes, assessing the

impact that they would have on response rate and

completion, respondent profile, and the way they answer

the questions

Qualitative exploration of the respondent experience when

completing the paper and electronic versions of the

questionnaire

This leads to recommendations on:

• Whether and how to implement the method changes from 2017

• Other improvements which might be made, to enhance the

respondent experience and or / increase volume and quality of

response

5

Detail on the method: the quant pilot

The Spring 2016 wave of the

NRPS went ahead as

normal:

• A representative survey on c.

30,000 passenger journeys

across all UK TOCs

• 12-page paper questionnaires

distributed to passengers about

to make a train journey

• 1,662 fieldworker shifts, taking

place between 11th January and

20th March 2016

Alongside the main NRPS we conducted 185 pilot fieldwork shifts

in which:

• Shorter versions of the questionnaire were used (equivalent to

8 pages)

• Respondents were offered the choice of paper or online

questionnaires: • In 75 of these respondents were invited to complete a set of „main‟

questions, followed by „optional‟ questions about stations and access

• In another 75, the same applied but with „optional‟ questions on fares

and ticketing

• In the final 35 fieldwork shifts, all of the same questions were asked but

none were presented as „optional‟ (this cell used the fares and ticketing

questions)

Fieldwork for these pilot cells was planned to match a sample of

fieldwork shifts in the main NRPS in order that direct

comparisons can be made between the main and pilot surveys, in

terms of response rates and survey findings

Hidden slide

6

Recruiting respondents in the main and pilot surveys

Fieldworker approaches passenger at station as they wait to board a train

Briefly explains purpose of survey and invites passenger to take part

Passenger takes a paper questionnaire and reply-

paid envelope

Passenger is offered option to complete on

paper or online

Completed questionnaire is posted back to agency

head office for validation and processing as normal

Passenger provides email address and receives

brief thank you note with a re-cap of the survey,

Transport Focus and the agency, and MRS details

Email address is sent back to agency head office;

personalised survey invite emailed to respondent

Completed questionnaire validated and processed

Main NRPS Pilot survey

Reminder email if needed

Hidden slide

7

The enhanced questionnaire (1)

Front page in colour, with

additional features

General update to design 8 pages in total

• 6 of „core‟ questions

• 2 of „optional‟ questions

Hidden slide

8


3 versions tested in the pilot – paper questionnaire shown; set up the same way in the online version

1 Station access qs, presented as optional Fare & ticketing qs, presented as optional Fare & ticketing qs, presented as part of

main q‟re 2 3

Hidden slide

9


2 completely new questions Hidden slide

10

Detail on the method: the qual evaluation

Teledepths conducted with those who had completed the pilot survey, and who

were willing to be re-contacted for further research*

• 10 x respondents who completed the paper questionnaire

• 10 x online via desktop computer

• 15 x online via smartphone

• In each group, a broad mix of demographics, TOCs used, journey purposes and other journey

details, and levels of satisfaction

* Note, these participants may inevitably have had higher engagement with the concept of the NRPS than „average‟ respondents

To evaluate the

experience of completing

the survey in each format,

and understand reactions

to particular aspects of the

enhanced questionnaire

and method

17 teledepths conducted with people who had started the online questionnaire

but not completed it (despite reminders)

• Broad mix of passenger and journey characteristics as above, plus

• Mix of those dropping out at different points

• Mix of those initially entering the survey on different devices

20 face to face user-test sessions with in-depth interviews, with people having a

range of visual, learning, social and cognitive impairments, testing a deliberately

‘accessible’ version of the online questionnaire

• A range of demographics, etc. as above

• All rail users but had not previously completed the survey

• Mix of those using different devices, and screen-reading software or other measures to aid reading

To explore where and why

people drop out in more

detail, in order to inform

possible improvements in

the future

To evaluate fitness for

purpose of this version,

and provide guidance on

potential improvements

should it be implemented

fully

1

2

3

Hidden slide

Overview of the pilot outcomes

12

Overview of the pilot outcomes

• The NRPS is important to passengers

as well as Transport Focus and the rail

industry

• On the whole the survey is straight

forward to complete

• …and questions are largely both

relevant and comprehensive

• Offering options to complete on

paper/online is a good move

• Shortening the questionnaire is logical

step; having variants to accommodate

missing questions works easily

• Offering missing questions as optional

appears to have little benefit

• New ‘softer’ questions add nuance –

perhaps relevant annually?

Of course, those giving qualitative feedback may have been „warm‟. But 19% of all respondents agreed to be re-contacted, and

even the drop-outs we interviewed largely agreed with the above, having specific issues which prevented their full completion

“Whilst the guy who spoke to me said I could do it online if I wanted to, I‟m an old

fashioned guy and I like paper that I can pick up and put down and I don‟t have to

re-boot the machine if I lose a page”

“She started telling me about the survey and everything, and I said well I haven‟t really

got a lot of time, so she said ok well I can send you a thing over the internet… she

asked me for my email address and … I think the survey came… within the same day”

“Generally speaking it was really easy to fill in. The questions

where I didn‟t have a huge amount to say were ones around the

security of the station just because it‟s never been an issue…but

the rest of it all seemed pretty relevant and easy to use”

“It‟s nice to actually be asked your view… I‟ve

used them for many many years, you don‟t

normally get to have your say in what goes on”

“I thought E-mail would be convenient for me in my case…I would have

been a little less inclined (postal), I didn‟t want to carry the paper home”

Online, desktop

Online, smartphone

Online, smartphone

Paper

Paper

Impact of the method and questionnaire changes on: Response rate

14

Response rates: pilot survey vs. control

Total Paper

(8 page q‟re) Online

No. recruited 8,830 12,000 9,193 2,807

Total recruits per shift 58.9 64.9 49.7 15.2

No. responded (pre-cleaning) 2,509 3,227 2,603 624

Total responses per shift 16.7 17.4 14.1 3.4

Response rate 28.4% 26.9% 28.3% 22.2%

Pilot survey Main survey

control sample* (12 page paper q‟re)

Comparable response rates: BPS / TPS

total paper online

BPS 33% 34% 25%

TPS 23% 23% 23%

• Offering a choice of completion methods* appears to result in increased recruitment

and therefore, despite overall lower response rate, a higher number of completed

interviews

• If online response rate can be increased, this benefit could be realised further

• Based on comparison of main survey control and paper pilot samples, the shorter

questionnaire does not appear to have encouraged greater rate of response

*this impact is likely to come from the choice of completion methods rather than the shorter

questionnaire: passengers are unaware of the questionnaire length at point of recruitment, and previous

pilot work on a 4-page questionnaire indicated this had little impact on recruitment rate

Total main survey (including FW

conducted on trains) = 30%

* The control cell comes from a sample of the main NRPS, for which the fieldwork (totalling 185 shifts) matched the

fieldwork conducted for the pilot in terms of the days, times and stations when / where passengers were recruited. More

details are given on page 5

15

Why agree to take part in the first place?

Importance of concept/duty

to other passengers

Belief that feedback = improvements

Fieldworker personality: Professionalism,

combined with „ordinariness‟ (though smartness = credibility)

Receiving survey quickly after

recruitment

“I thought I get the train quite a lot, and I care about

letting them know about my experience, if I could

help the public transport industry then I should”

An environment conducive to completing a survey - Several paper respondents did the survey then and there on the train; and several online respondents would have liked to.

- Online respondents often felt the online option allowed them to complete when convenient, and having it in inbox would mean they‟d remember it

“He did it in a professional and courteous way…he

seemed like a nice polite man”

“I think it‟s important that we give feedback, things can only

be improved by people giving constructive feedback as

opposed to being critical all the time”

“I was actually pleasantly surprised at how

quickly it came…”

Perceived ease of participation

“When I‟m approached, there‟s always a split second of heart-sinking,

but … he was quite charming, so I just thought „yeah, alright, why not,

I„ve got the time‟… he had a good manner”

“A big thing was that the envelope was already there, so I knew I

wasn‟t going to have to go and pick up a stamp anywhere or post it

into the station, I could just post it in a post box when I passed one”

Online, desktop

Online, desktop

Online, desktop

Online, desktop

Paper

Paper

16

Are we making enough of the NRPS concept?

• Many depth respondents skimmed/ignored

introduction at start of the survey

• Many reported hearing only a little about the

survey purpose from fieldworker – but this

was felt to be appropriate given the time

constraints

• Focusing on TF during intro is not

particularly beneficial

“The guy … said it was a „passenger survey‟… I knew it was something

to do with giving our views rather than them advertising something…

„passenger survey‟ was enough to grab my attention ”

“Only that it was to try and improve the service; because she knew I

hadn‟t got much time she didn‟t really waste a lot of time talking about

the background – but fine, that was enough to pique interest”

“Perhaps worth telling people that the results will be made public. I

think if people think it‟s some secret deal between the train

companies, it‟s frustrating”

…But respondents are interested and

motivated by the survey purpose (and are

also interested in seeing results)

Keep fieldworker intro and questionnaire ‘blurb’ short, but with focus / emboldening on key words and phrases / concepts:

• Passenger views / passenger survey / passenger feedback

• Independent

• Transparent / public (but avoiding any inadvertent implication of non-confidentiality)

• To inform service improvements

Paper

Paper

Paper

17

Response rate time between email invite and recruitment (days)

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

0 1 2 3 4 5 6 7 8

Response rate Proportion of all recruits

Days between recruitment and email invite

Resp

on

se r

ate

Sample

size:

561

Sample

size:

1238

Sample

size:

203

Sample

size:

16

• Relatively few respondents were invited to take part in the online survey 3 or more days after initial recruitment; as a result

response rate analysis after this point is somewhat unreliable

• Taking into account only those respondents who were invited within 3 days of recruitment, response rate is clearly stronger

the sooner the survey invite is received

• As with BPS, the aim must be to send out survey invites within a day of recruitment, ideally on the same day

18

Relative contribution of online and paper respondents

Total contribution to

recruitment

Total contribution to final

results

Paper 77% 80%

Online 23% 20%

Of which…

Recruited via tablet 49% 50%

Recruited via paper form 51% 50%

• Relatively small contribution from online respondents in this pilot – though higher than seen

in smaller NRPS pilot in Spring ’15 (18%), and BPS (7%), and similar to TPS (23%)

• Measures to increase response rate among those starting online are recommended in this

report, to improve contribution of online in future

Hidden slide

Impact of the method and questionnaire changes on: Who takes part

20

Profile of survey respondents

Total Paper

(8 page q‟re) Online

Commuter 39 36 35 43

Business 15 15 15 16

Leisure 46 49 51 40

16-34 23 23 20 38

35-54 36 38 37 38

55+ 39 37 41 20

No answer / prefer not to say 1 2 2 3

Male 43 43 42 48

Female 55 52 43 48

No answer / prefer not to say 2 5 5 4

Weekday 84 84 84 85

Weekend 16 16 16 15

Pilot survey Main survey

control sample (12 page paper q‟re)

• Despite slightly higher incidence of leisure journeys in this pilot (vs. control), introducing an online element has potential to increase

representation of commuters and (probably overlapping with this) younger males – IF online contribution can be increased

• NB in a live survey, data would be weighted by journey purpose so ultimately survey profile would not alter. Rather, level of weighting

required should be lessened

Impact of the method and questionnaire changes on: (Satisfaction) results

22

Key results in the pilot vs. control surveys

Main survey

control sample Pilot

Headline satisfaction ratings

Overall journey satisfaction 84 82

Overall station satisfaction 82 83

Overall train satisfaction 80 79

Value for money rating 48 50

Summarised station ratings

Station environment 71 73

Station facilities 68 56

Transport / connections 62 64

Customer service 79 81

Summarised train and journey ratings

Journey measures 79 79

On-board environment 70 67

On-board facilities 50 53


Very slightly lower journey satisfaction

overall in pilot – but no clear pattern,

suggestion little impact on results if this

method was implemented fully

• Influence of more negative online

respondents (see next page)?

• Attention drawn to more specific aspects of

on-board experience (crowding, wi-fi, power

sockets) – which we know is more influential

on overall satisfaction than station

experience?

VFM and station satisfaction actually slightly

higher in pilot

• Slightly higher proportion of leisure

journeys? (commuters could be increased

with stronger online contribution, but either

way would be controlled for in a live survey

by weighting)

• New focus on additional station facilities

(toilets and w-fi) less impactful than

equivalent new questions re on-board

experience, due to lower overall importance

23

Online vs. paper responses in the pilot survey

Paper Online






Summarised station ratings

Station environment 73 73

Station facilities 58 50

Transport / connections 65 60


Summarised train and journey ratings

Journey measures 80 75

On-board environment 68 61

On-board facilities 51 44


People responding on paper are fairly

consistently more positive than those

responding online

Partly a function of online respondents’

age and travel behaviour (more likely to

be commuters):

• However, analysis* suggests that, while

the pattern isn‟t strong, on balance there is

a little more negativity among online

respondents, even when controlling for

age and journey purpose

• This echoes recent findings for BPS and

TPS which also suggested that online

respondents are generally a little less

positive overall

Positivity does not appear to be linked to

any time delay in response:

• Demonstrated in BPS, and the pattern of

more negativity exists here in NRPS,

where the time delay has been reduced

dramatically compared to BPS

• Online/paper differences would be partly

mitigated by journey purpose weighting,

but if online respondents make more

contribution in the future we might expect

this to bring about a slight suppression of

satisfaction results.

*Not shown in this summary report; available if required

24

So, we have applied journey purpose and weekday weights to assess impact on a more level playing field:

Pilot Weighted sample

profile

Commuter 39 36 46

Business 15 15 15

Leisure 46 49 40

Main survey

control sample (12 page paper q‟re)

All weighted to…

Proportion of commuters in the sample increased in both pilot and control samples

Main survey

control sample Pilot






• All headline results are suppressed a little

in both samples as a result of up-weighting

commuters / down-weighting leisure

travellers

• Where differences existed in the un-

weighted results for station and VFM rating

(where pilot results were more positive,

these are flattened

• But other measures still seeing more

negative results in the pilot

• Overall the trend for slightly more negative

response in the pilot remains, even after

weighting (though again, pattern is not

100% consistent, with higher scores in the

pilot for some station measures)

Effective sample size reduces

to 96% of un-weighted base

in the control, 98% in the pilot

Impact of the method and questionnaire changes on: ‘Missing’ questions

26

Around 6-8% drop out from the paper survey when given the „choice‟

3 4

2 4

2 1

8

11 12 12 12

11

3 4

2 2 3

1

5

9

15

9 9 9

4 2

1 3

1 3

8

3 4

5 4 4

0

5

10

15

20

25

30

35

Paper: F&T questions - optional Paper: Station questions - optional Paper - F&T questions - mandatory

Questions (selected)

% n

ot a

nsw

erin

g q

ue

stio

n

NB. Age and emotions questions asked after

fares / stations sections in mandatory version

Questions offered as

optional from this point in

„optional‟ versions

Impact of „optional supplements‟ on continued response: paper

27

…but a quarter of online respondents drop out with the same offer

0 0 0 0 0 0 2

30 30 30 30 30

0 0 0 0 0 0 1

28 28 28 28 28

0 0 0 0 0 0

4 3 3

4 4 4 0 0 0 0 0 0

6

3 4 4 4 4

0

5

10

15

20

25

30

35

Online: F&T questions - optional Online: Station questions - optional

Online: F&T questions - mandatory Online: Station questions - mandatory


% n

ot a

nsw

erin

g q

ue

stio

n






Impact of „optional supplements‟ on continued response: online

28

…and as expected this is a little more pronounced among smartphone respondents

0 0 0 0 0 0 2

30 30 30 30 30

0 0 0 0 0 0 1

28 28 28 28 28

0 0 0 0 0 0

4 3 3

4 4 4 0 0 0 0 0 0

6

3 4 4 4 4

0 0 0 0 0 0

3

34 34 34 35

34

0 0 0 0 0 0 2

0 0 0 0 0 0

5

10

15

20

25

30

35

Online: F&T questions - optional Online: Station questions - optional Online: F&T questions - mandatory

Online: Station questions - mandatory Mobile: F&T questions - optional Mobile: F&T questions - mandatory


% n

ot a

nsw

erin

g q

ue

stio

n






Impact of „optional supplements‟ on continued response: online, completed on mobile phone

Hidden slide

29

• Might as well – have got this far

• My feedback is important

• Can see it‟s not much more/looks easy

• Experience so far has been fine

• Missed the fact it was optional (didn‟t read the blurb)

• NB. a minority indicated that the offer to finish „early =

sense of respect for respondent

Why do so many respondents continue with optional questions?

“Overall the survey is good….partly because I started it, I hadn‟t got

bored, it‟s important, also respect…we‟ve gotten this far…I want to do it”

“I suppose in the hope that I would be able to say something

more general about the service, because that specific journey

was absolutely fine and that was not indicative of my

experience with that company all the way along”

“I look at that bar at the top right hand corner, the

fact that it said 85% was okay for me”

“Just to see if there was anything that I thought

wasn‟t included”

We recommend not presenting some questions as optional

• Presenting as optional encourages drop-outs among certain groups – with

potential for minor skews in results

• But presenting as mandatory has no material negative effect

…but having variants to enable a shorter questionnaire seems sensible

“Because you had won me over [by that point] and I

trusted the survey and probably because I still had a

few minutes to spare! Also it was all relevant, and

because I do so much travel I thought well hold on,

no, these guys are trying to make travel better so

let‟s give them my two penn‟th”

“My answers could make a difference so it seems

a bit silly to not bother. I mean if it had been an

extra 10 pages … but an extra couple on top

didn‟t seem like too much for me to do”

Online, desktop

Online, desktop

Paper

Paper

Online, desktop

Online, desktop

The softer side of passengers‟ experience: New questions on trust and how people feel

31

How does a train journey make passengers feel?

Relaxed

Happy Indifferent

Worried

Stressed

Bored

Angry

Frustrated

29% 32%

16% 12%

% choosing each emotion

No answer: 6%

1%

1%

2%

1%

32

Which passengers feel positive and negative?

Positive Neutral Negative

A little older (45% are aged 55+ compared to 37% on average)

A little younger (34% are aged 16-34 compared to 23% on average)

More likely to have disability (19% vs 12% on average; specifically, these passengers are

more likely to feel worried/stressed/frustrated than angry)

High proportion travel for leisure (61% vs 49% on average)

High proportion commute (47% vs 36% on average)

High proportion commute (55% vs 36% on average)

High proportion delayed (32% vs 24% on average)

Very high proportion delayed (53% vs 24% on average)

More likely to be affected by anti-social behaviour (24% vs 7% on average)

Naturally, experiences related to or caused by TOCs impact greatly on how passengers feel.

But TOCs are not the only influence:

• Behaviour of other passengers (….but can TOCs help via enforcement of guidelines?)

• External factors and journey context (…but can TOCs take extra care at crunch times to make people feel better e.g. with Tube-style comms / interaction?

• Passengers themselves – demographics and personal circumstances

Hidden slide

33

% p

assengers

left

feelin

g n

egative

Overall journey satisfaction (fairly/very satisfied)

Any negative emotion* vs overall journey satisfaction (fairly/very)

Any positive emotion* vs overall journey satisfaction (fairly/very)

% p

assengers

left

feelin

g p

ositiv

e


Positive emotions and overall journey satisfaction correlate well

% p

assengers

left

feelin

g n

eutr

al


Any apathetic emotion* vs overall journey satisfaction (fairly/very)

Strength of correlation between

overall journey sat and…

Relaxed 0.79

Happy 0.74

Indifferent 0.70

Bored 0.52

Angry 0.48

Frustrated 0.08

Stressed 0.04

Worried 0.00

(R² values)

Bored vs dissat: R² = 0.53

Worried vs dissat: R² = 0.02

*relaxed / happy

*angry / frustrated / stressed / worried

*indifferent / bored

Strong inverse correlation between journey satisfaction and

indifference in particular suggests that satisfactory journeys

genuinely leave passengers feeling pretty good (when we ask

them to think about it!), rather than simply being ‘fine’ as we have

sometimes speculated previously for the meaning of ‘satisfaction’

34

% p

assengers

left

feelin

g n

eutr

al


Any apathetic emotion* vs overall journey satisfaction (fairly/very)

Any positive emotion* vs overall journey satisfaction (fairly/very)

% p

assengers

left

feelin

g p

ositiv

e


% p

assengers

left

feelin

g n

egative


Any negative emotion* vs overall journey satisfaction (fairly/very)

…but negative emotions have much weaker relationship with satisfaction

Strength of correlation between

overall journey sat and…

Relaxed 0.79

Happy 0.74

Indifferent 0.70

Bored 0.52

Angry 0.48

Frustrated 0.08

Stressed 0.04

Worried 0.00

(R² values)

Bored vs dissat: R² = 0.53

Worried vs dissat: R² = 0.02

*relaxed / happy

*angry / frustrated / stressed / worried

*indifferent / bored

35

Do the emotions questions add value?

Total Very

satisfied

Fairly

satisfied

Neither

/ nor

Fairly

dissatisfied

Very

dissatisfied

Relaxed 16 28 13 3 - 2

Happy 29 51 23 3 1 8

Indifferent 32 14 47 48 15 -

Bored 12 1 9 34 52 24

Worried 1 0 1 3 6 -

Stressed 1 - 0 2 4 5

Angry 2 0 1 2 14 39

Frustrated 1 0 0 1 6 15

“It was about my

expectations. I want the

train to be on time, to be

clean, and it was that so

it was ok, but it wasn‟t

exceptional, it was what I

would expect”

(Indifferent, fairly satisfied

overall)

Although positive emotions simply tell a similar story to the „overall journey satisfaction‟ question, negative

emotions, though rare, highlight other elements of passenger experience

TOCs are generally quite good at making people “fairly satisfied” – but this often means they‟re left feeling indifferent.

This may be the maximum, and appropriate expectation for many passengers, but perhaps TOCs can still work

harder to leave people feeling good even when they do generally satisfy on a basic level?

Plus, qualitative feedback indicated that…

• Greater variety in question format/presentation may boost engagement

• Interpretation of images is fairly consistent and as intended

36

Do passengers trust train companies?

% passengers trusting TOC travelled with today

Do not trust

them at all

Trust them a

great deal

1 2 3 4 5 6 7

No answer: 3%

3% 6% 7% 15% 23% 27% 15%

37

Which passengers are more / less trusting?

Trust a

great

deal

% o

f th

ose s

corin

g

each le

vel

Do not

trust at

all

Journey purpose: commuters find it much harder

to trust their TOC than leisure travellers

In addition, the most trusting people are typically:

• Older

• More likely to be female

• Less affected by anti-social behaviour

• On longer journeys, including those involving a reserved seat

…and unlike emotions, trust appears to be impacted little by disability

Occurrence of a delay

Strongest profile variation is seen with…

55

74 84

1-3 4-6 7

Score out of 7

% experiencing no delay during today‟s journey

Hidden slide

38

Trust and overall journey satisfaction are correlated a little less strongly than positive emotions

Tru

st (n

et th

ose s

coring 6

-7 o

ut of 7)


Overall trust (6-7) vs overall journey satisfaction (fairly/very)

Tru

st (n

et th

ose s

coring 7

out of 7)

Overall journey satisfaction (very satisfied)

Absolute trust (7) vs very satisfied with overall journey

(Absolute trust (7) vs overall journey sat: R² = 0.7805)

Trust is about general experience and overall opinion, and

so this question does say something different – though

not contradictory – to the overall satisfaction question

Consider asking about trust on an annual (rather than bi-

annual) basis?

Evaluating respondents‟ experience of the survey to increase response

40

Overview of experience in completing the survey

1 Generally straight forward

2 Questions largely relevant and

easy to understand and answer

3

Presentation, on paper, desktop

and mobile, generally either not

mentioned (so no issue) or

complemented

4 For some, survey is quite long

“Questions that are succinct and straight to the point”

“Text size was good, clear where you had to put

responses in”

“It took a long time to get a quarter done”

“Clean, clear”

“Well I didn‟t have any problems filling it in… I think it was clear

what you had to do, I don‟t think anyone would get confused… I

didn‟t find it difficult – sorry I‟m not being very helpful am I!”

“It was quite a long and meaty questionnaire – I suppose

you have to do all the extra ones like ethnicity?”

“You don‟t bang on about household, kids, household income…black,

white, disabled or whatever, which often takes up so much survey

space. I think it was tailored specifically to rail, it was good ”

Paper

Online, desktop

Paper

Paper

Online, desktop

Online, desktop

Online, desktop

Evaluating respondents‟ experience of the survey to increase response: The presentation of questions

42

Summary of feedback on the questionnaire

Generally, a favourable or „no issue‟ reaction to layout and presentation

• Clear, looks straightforward

• Not explicitly praised as „engaging‟…but

not criticised as daunting or very boring

(like the old version)

• Professional, credible looking

• Clear, clean

• Straightforward and easy

• Not „exciting‟ (could more colour and questions layouts

be used?)

• …but bear in mind consistency = fast + easy to complete

• Few, and minor, specific issues for mobile

Improving the online survey experience

44

Drop out points through the survey, by completion device

% o

f sta

rters

still

in s

urv

ey

Proportion using device at:

desktop tablet phone

Survey start 43% 13% 44%

Overall journey sat 44% 15% 42%

Emotions 44% 14% 42%

Desktop and smartphone users drop out

of the survey more readily than tablet

users (but since they drop out at a similar

rate, overall device profile of respondents

changes little from start to end)

Faster survey invites in future could push

more survey completion onto tablet (due

to preference for completion during

journey), aiding retention

45

Drop out points through the survey: culprit questions (1)

% o

f sta

rters

still

in s

urv

ey

Immediate drop out with date / time

question – necessary for QA, and to

set context for respondent at outset

Scheduled departure time

Drop between antisocial

behaviour and age question:

qual evaluation indicates

open-ended question is the

culprit, rather than age

46

Drop out points through the survey: culprit questions (2)

% o

f sta

rters

still

in s

urv

ey

Overall station satisfaction,

followed by journey ratings

On-board train ratings

Overall train satisfaction

Overall journey satisfaction

Evaluating respondents‟ experience of the survey to increase response: Questionnaire content and scope

48

“Your journey today”

The “any other comments” question

is valued

No issue with understanding that the

questions relate to today‟s journey

…but many are concerned that we might miss more

typical feedback

” “

Most of the questions will be about the

journey you were making when

approached by the interviewer. However,

if you would like to feed back about more

general experience, there will be

opportunity for this

“I liked the overall opinion question… say for example on a

normal Saturday I might have got into London and everything

was fine,… but my overall experience of the company [due to

weekday commute] is not great so it‟s good to give people a

chance to say, yeah this journey was great but in general there

are other things here which should be noted”

Could we let respondents know up front that they will

have this opportunity, both to appease frustration for

those who do take part, and potentially limit drop-out? “The only thing I‟d like is to make more comments

about my general experience … to compare this

journey with others… the context, I think that is

important to people… it takes quite a long time to do

that survey and it‟s quite frustrating… I understand

about sampling…but you have to do it on people‟s own

terms if they‟re giving their time… it‟s very important

that people feel you are listening to them”

“I think it was pretty clear, the survey is quite specific

when it wants you to think about your journey „on this

day‟… the questions direct you quite straightforwardly to

whether you need to be thinking about kind of an overall

view or this particular occasion”

Paper

Paper

Paper

49

The survey length

Despite the shortened pilot version,

questionnaire length is still a common

criticism

(and this is among those who have completed it and agreed

to follow-up research)

(…It‟s not all bad, a few of our (warm)

respondents had no issue with length,

particularly on the paper version)

What can we do about it?

• Really clear signposting upfront and throughout?

• Improvements to progress indicator (especially for

grid questions)

• Additional variety within question format (within

reason)

• Clearer notification that you can save and return

later

(Suggestions are based on feedback from respondents and our

observations, rather than direct suggestions from respondents

themselves)

This survey will cover:

• Your journey today

• Your experience at the station where you started

• Your experience on board

• …etc

“Maybe there‟s too many questions…maybe the

survey is a bit on the long side”

“Shorter…it was a big thick document and if I‟d

have seen that [before I accepted it] normally

I‟d have said no, and other people even said to

me, „you‟re brave taking that on”

“It wasn‟t too long, the print was clear. Black ink on a white

background, it was a lot easier to read than some fancy

surveys I sometimes get from other people”

” “

Paper

Paper

Online, desktop

50

Does it matter that some people complete the survey during their journey?

Of all 26 core questions in the shorter version of the

survey, in normal circumstances only 5

questions/statements within questions cannot be

answered until towards end of journey

• Punctuality

• Whether got a seat for all/part of journey

• Whether delayed

• Several Qs on how well delay was dealt

with (typically relevant to c. 25%)

• Overall satisfaction

…and in most cases passengers have a very good idea

of how to answer, within their first few minutes

If they‟re taking the time to provide feedback,

passengers want their feedback to be accurate

Anecdotal feedback suggests response rate

would drop off dramatically if post-journey

completion was enforced (if it could be!)

Recall of journey details and experience (especially if emotional Qs are included) is

immediate if survey is completed then and there

“…But there were a couple of questions that I left…

about the journey itself that I wasn‟t able to answer,

so I did a couple and finished it off when I got to work

… it was the stuff about the train itself, but all the stuff

about the station, I do it every day so it was easy

enough to answer those ones [while waiting for the

train in the first place].” “Sorry… if the answers were messy! – I had to go back and

cross them [information provision] out because when we were

delayed they suddenly went quiet”

“I think I‟ve been handed them before, and I‟ve just dropped

them in my bag and haven‟t done anything with them, but

[because I was delayed by 20 minutes] in this case I had time”

We strongly recommend no alteration to the current notional

agreement, that respondents should be guided towards completing

after their journey, with acceptance that many may not

Paper

Paper

Paper

Evaluating respondents‟ experience of the survey to increase response: An accessible survey

Hidden slide

52

Re-cap on background

• BDRC Continental created a more accessible version of the online

NRPS questionnaire, with enhancements to aid participants with

some visual impairments or certain types of learning difficulties

• Participants with a range of visual impairments or learning difficulties

tested the survey on their own device and gave feedback on both

specific questions and the survey as a whole

Transport Focus are looking for ways to make it easier for

as many passengers as possible to take part in the NRPS

Methodology

• 20 individual and paired in-depth interviews with passengers

• Conducted face to face, with the interviewer observing the participant as he or she tested the accessible online

questionnaire

• The participants tested the survey on their own device that they would normally use to access the internet

• Fieldwork took place in March and April 2016

Hidden slide

53

Difference between the „accessible‟ and „standard‟ surveys

• The main changes to the questionnaire design were:

• Larger font

• Black and grey text on white background – minimal colour

• Question format for the station and on train ratings changed

to drop down boxes

• Vertical instead of horizontal scales

• Text underneath pictures (emotions question, logos)

Hidden slide

54

Overall, having an online survey is better than just offering the paper version

The survey has the greatest impact for people with complete or very severe blindness (e.g. those using

screen-reading software or those who need to enlarge the screen dramatically)

Strange enough, I think if someone sent me a link to

my email I would be much more likely to do it. If

someone took my email address and said can we

send you something, I am much more likely to do it

because then for me personally doing it on the screen

is much faster, it‟s easier, better to see and you press

the submit button and feel good because it‟s done

- Desktop user, enlarging text

On quite a few surveys I‟ve tried, because of the

access issues on some of them I must admit I‟ve given

up and thought it‟s not worth it

- Desktop user, JAWS/NVDA software

Overall feedback on the survey

Very accessible for me, I went through it quite quickly, works well with

JAWS…I felt I was able to do it as well as anybody else

- Desktop user, JAWS software

The survey reads well, and works better with

screen-reading software than a lot of other

websites and online surveys

Easy to do, the survey design is consistent

Respondents would choose an accessible survey

over a non-accessible survey and appreciate being

given the choice

The survey is a bit too long, as more effort is

required for these respondents, when enlarging

the screen or using screen-reading software

I think it covered everything

- Desktop user

Hidden slide

55

Feedback on the survey content

On the whole, the survey is felt to cover all areas of the passenger

experience

?

Some topics are irrelevant to a lot of respondents with visual

impairments. While these cannot be taken out of the survey, the

respondents are missing a „not relevant‟ option. This could

perhaps be added as the first option on the drop down menu

The survey is felt to be too long - the overall length is exaggerated

by screen-readers and the extra effort required (e.g. enlarging the

text)… can we be really ruthless with the questionnaire for the

minority completing this version?

Hidden slide

56

We recommend offering a ‘standard’ and an ‘accessible’ version on the very fist screen, with guidance

as to who the ‘accessible’ version is relevant to (i.e. those using special software or enlarging

dramatically). Participants can then choose the one they think would be best suited for them

Alternatively the first questions could be a screening, including a question on disability. The accessible option

could then be offered to those who may benefit from it

Generating take-up

The respondents appreciated being made aware of the existence of an accessible survey, as

some would dismiss the idea of taking part on the assumption that it would not be set up to work

for them

Having an accessible survey:

How to alert people to the fact?

makes them feel like valued passengers, which in itself encourages

response

creates positive reflection on the rail industry as a whole

But this is very difficult to do, and not always appropriate (especially for

those with less visible impairments and learning/cognitive difficulties) ?

The respondents we talked to would like to be alerted to the existence of

an accessible survey at point of recruitment

Hidden slide

57

Is it worth it?

• Some participants feel strongly that the „standard‟ version

should be accessible, rather than having two versions

• The accessible survey only meets the needs of a niche:

Those who use screen-readers

Those who enlarge the screen dramatically

• For most, drawing attention to an accessible survey has

benefits that outweigh the problem of “being different”

• By offering an accessible survey, you encourage these

passengers to take part

• It also creates a more positive and inclusive image of the rail

industry as a whole

• There would be too many compromises in a merged version,

which would then not truly benefit those it should (e.g. the

drop down menu works well with screen-readers, but the

„standard‟ version is visually engaging and encourages

response amongst passengers without visual impairments)

We recommend maintaining two separate surveys,

and offering the accessible version to those with

complete or very severe blindness

Participants with less severe visual impairments

usually find the ‘standard’ version easy enough to

use and more engaging

It needs to be made clear to respondents who

would benefit from the accessible survey, to avoid

more people using this option than needed

Hidden slide

Summary…and what next?

59

Key take-outs (1)

• NRPS still gets a good review from passengers – in terms of purpose and overall respondent experience

Dual data collection

• Offering both paper and electronic completion options does bring benefits:

o Broadens appeal and convenience to more passengers

o …in particular to visually impaired people

o Enables better completeness of response (fewer missed answers / mistakes)

o Has potential to increase representation of commuters and younger people, improving weighting

efficiency – and survey credibility

o Overall a dual approach enables higher volume recruitment = more efficient fieldwork

And increasing the contribution of online response could realise these benefits further – gains already

made since learnings transferred from BPS Autumn ‟15

• Note of caution: an online element does have potential to supress satisfaction ratings a little, but:

o We recommend the benefits of greater representation (a closer picture of reality!) outweigh this

o Weighting by journey purpose will control the effect, which is likely to be marginal – paper likely to

remain dominant over online for the foreseeable future

60

Key take-outs (2)

Questionnaire enhancements: shortening the survey

• Though difficult to completely separate the impact of data collection vs questionnaire enhancements, a

shorter questionnaire does not appear to have contributed to response rate or quality (unlike a much more

ruthless 4-page option might)

• …however, logically, a shorter questionnaire is more palatable for respondents, and creating variants to

achieve smaller (but still robust) samples for non-core questions is relatively straightforward

We recommend implementing the 8-page questionnaire with variants

• Deliberately offering non-core questions as „optional‟ is appreciated by a minority, but by this point in the

survey most are happy to continue

• …and while minor, there is potential impact on results from these questions

Overall we recommend not offering questions as ‘optional’, and treating the full 8 pages as standard

61

Key take-outs (3)

Questionnaire enhancements: the softer side of passenger experience

• Adding a new trust question has mixed outcomes:

o Adds a different layer to understanding passenger relationship with rail industry

o Allows respondents to express broader opinions – which is valued

o Disrupts and re-engages respondents

o …but confuses some, and in extreme cases acts as „last straw‟ if other questions have felt irrelevant or

overall too long

On balance we suggest it’s worth keeping – perhaps on an annual basis given sentiment

unlikely to change quickly

Merits of the emotions question – as part of NRPS – are less clear:

o Positive emotions add relatively little to overall journey satisfaction

o But negative emotions can help highlight issues more keenly than the satisfaction rating, with a more

personal flavour

o Generally popular with respondents – good for overall engagement and survey image

Is NRPS too blunt an instrument for effective capture and use of emotions data (at least on a

regular basis)?

62

Key take-outs (4)

Optimising response

• Response to the survey continues to be driven by:

o Opportunity for and perceived importance of feedback

o Engaging interaction with fieldworkers

o Measures to reduce respondent effort

….which must be front of mind in all development / enhancement of the survey going forward

• And we suggest some enhancements to improve respondent experience, to maintain engagement

and boost response:

o Further tweaks to early questions to reduce effort

o Up front sign-posting of survey content, especially the up-coming opportunity to give general (not just

today‟s journey) feedback

o Indication of average completion time for online survey

o Various small programming / formatting tweaks to reduce niggles

o Full implementation of „accessible‟ survey (with some programming improvements)

Thank you….

Enhancements to the National Rail Passenger Survey€¦ · paid envelope Passenger is offered...

Documents

Transcript of Enhancements to the National Rail Passenger Survey€¦ · paid envelope Passenger is offered...