The comparability of Don’t Know answers between CATI and ...€¦ · The CAWI questionnaire...

The comparability of Don’t Know answersbetween CATI and CAWI modes

Kirsti PohjanpääMarjaana Järvensivu (Survey Laboratory)

Statistics FinlandESRA conference 16.-19. July 2013

The contents of the presentation

1. DKN option in the web questionnaire of LFSQualitative pre-testNov 2012

2. DKN option in the web pilot study of LFSsome plansOct 2013

3. DKN option and answers in the mixed mode pilot study ofConsumer SurveyCAWI vs. CATINov 2012

16th July 2013, ESRA 2Pohjanpää, K. & Järvensivu, M.


Consumer Survey and Finnish Travle Survey,2nd pilot (WEB04), Tiina Heikkinen

Plans

Results

Results

Developing web data collection in personal surveys

Part of ESSnet project DCSS

The projects of LFS are part of Eurostat's ESSnet projectData Collection for Social Surveys using Multi Modes2012–14German, the Netherlands, UK, Norway, and Finland

http://www.cros-portal.eu/content/data-collection-social-surveys-using-multiple-modes-dcss


The Web Questionnaire of LFSPre test

16th July 2013, ESRA A 5Pohjanpää, K. & Järvensivu, M.

Background

In the Finnish Labour Force Survey (LFS), data is collectedby telephone interviews (CATI mode).In few years, also CAWI mode will be available forrespondents.Data collection mode may influence survey results

When combining CATI and CAWI modes, it’s significant tofind out if any mode effects will exist and to assess how tohandle those effects.

qualitative pre-testing, pilot studies…

616th July 2013, ESRAPohjanpää, K. & Järvensivu, M.

Don’t know option (DKN)

In the CATI mode all questions need to be answeredEmpty options not allowedIf the respondent does not know or does not want to givean answer, the interviewer can use codes “don’t know”or “refuses to answer”.However, these options are never read aloud for therespondent, andthe options are needed very seldom (max 1%)

The Finnish LFS is based on individual sampleThe share of proxy-interviews is very low (4%).


Don’t know option

The CAWI questionnaire follows the same idea; emptyoption not allowed some exception

Thus, in every question a respondent is forced to answer(at least Don’t know)As the LFS questions are rather straightforward, theproblem with the DKN is not that respondents would notknow the answer but rather that they might not bemotivated to respond.


Pre-testing the Don’t know option in CAWI mode (1)

The aim of qualitative pre-testing is to find comparable way topresent the DKN option between the CATI and CAWIquestionnairesTo achieve maximal functional equivalence we tried to imitatethe structure of CATI mode in CAWI as much as possible

1. All questions are obligatory, and2. the DKN option is hided.3. If a respondent tries to pass the question without

answering, the DKN option with an error messageappears on the screen


Pre-testing the Don’t know option in CAWI mode (2)

Implementation of qualitative pre-testing22 pre-test interviews (Nov 2012 – Mar 2013)Methods used: cognitive interviewing and observationSurveyLaboratory http://stat.fi/tup/surveylab/index_en.html


Main results of Pre-testing

Respondents seldom needed the DKN in CAWI mode,at least in the pre-testing environment.

Respondents were motivated to answer thoroughlyVery few respondents said that they would have needed aDKN option in any of the questions

=> Our solution seems to be working in the LFSto show the DKN (+ an error message) only afterrespondent tries to skip a question

It will be tested in the web pilot of LFS


The Web Pilot of LFS 2013Some plans


The Web Pilot Study of LFS

Sample size n=8,000A random sample of 15 to 74 year-oldsData collection at Oct 2013Only 1. waveOne aim is study the mode effects (CAWI vs. CATI)LFS normal data as a comparison data

data collection at the same timedifferent data collection mode (telephone vs. web)

There will some split-plot tests in the pilot, one of them isconnected to the DKN option.


DKN option as a split-plot-test, two groups


Group A Group B

Don't know option In view If needed

Sample size n=4,000 n=4,000

The Web Pilot Studyof LFS, plans

The DKN option in view (group A)


I don’t knowEn osaa sanoa



The DKN option shows only if needed (group B)

If a respondent clicksthe Continue-buttonwithout answering…

No DKN option



The DKN option shows only if needed (group B)

…the DKN optionwith an error messageappears on the screen

”Please, answer to this question orchoose the ’Don’t know’ option."


The Mixed Mode Pilot Studyof Consumer Survey 2012


The Mixed Mode Pilot Study of Consumer Survey

A random sample of 15 to 84 year-olds, n=2,350Data collected at Nov 2012First web (10 days), after that a telephone interview (for nonresponse)An advance letter + one reminderOne target of the pilot was to study the mode effectsResponse rate 55.6% (Consumer Survey 61.1%)

Responses n=1,305Web 23.5%


DKN option in the pilot study

DKN option was offered as one alternative in CAWIDKN option was in parentheses

YesNo(Dont’t know)

Different from the telephone interviewthe DKN option was not mention at all


The Mixed Mode Pilot Studyof Consumer Survey

”How do think the economic situation in Finland istoday compared with one year ago”

A lot betterA little betterThe sameA little worseA lot worse(Dont’t know) in CAWI




Don’t know answersThe Mixed Mode Pilot Studyof Consumer Survey

Finnish economy, now

The prices next year

Unemployment next year

Own economy, next year

0 3 6 9 12 15

CAWI CATI

The fact questions and DKN; questions about the property

There were several questions about the property of theelectronic stuffs of the household in Consumer Survey (n~30)

Some differences between CAWI and CATI in DKN

In average, no big differences:CAWI 1.04% to 3% DKN answers

CATI 0.81% to 2%Yet, some questions were difficult to answer in web…



“Does your household use TV connection thatworks through broadband, i.e. IPTV?”

INFO: IPTV is interactive TV technology that worksthrough broadband connection, e.g. Elisa Viihde, SoneraKoti TV

The info button is beside the question, and you mustclick it to see the contents

IPTV = Internet Protocol Television


Don’t know answers the biggest



IPTV

WLAN

tv by PC

0 3 6 9 12 15CAWI CATI

Summary

The DKN option is prolematic in mixed mode data collectionThe situation is different between fact and opinion questions

Advices, and additional information play an important rolein the process

We hope that our idea of DKN will workThe results of the web pilot of LFS are available 2014

Thank you!


The comparability of Don’t Know answers between CATI and ...€¦ · The CAWI questionnaire...

Documents

Transcript of The comparability of Don’t Know answers between CATI and ...€¦ · The CAWI questionnaire...