Post on 11-Jan-2016
Using survey data collection as a tool for improving the survey
process
Silvia Biffignandi, Antonio Laureti Giulio Perani University of Bergamo Istat Istat
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Outline Outline
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Introduction
Survey
Questionnaire
Preliminary results
Research in progress
Introduction: Paradata and web surveysIntroduction: Paradata and web surveys
Web surveys allow for the collection of paradata during web questionnaire completion;
paradata are data generated during the fieldwork of the survey.
Two types of paradata: server-side paradata are collected by software tools running at the server. They relate mainly to the questionnaire compilation process. This data are contained in the so called logfiles.
client-side paradata describe how the respondents are answering the questions (order, questions skipped, keys that have been pressed and so on). (Biffignandi and Bethlehem, 2012; Biffignandi, 2010)
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Analysis of paradata for:
questionnaire improvement (identification of the most difficult questions in the survey form, of some missing (or too restrictive) checks )
quantification of the burden on respondents (from the degree of ready availability of R&D data in enterprises, to the time needed to fill in the questionnaire )
timeliness? costs? identification of the characteristics of late respondents.
Using survey data collection as a tool for improving the
survey process
Introduction: Tasks of paradata Introduction: Tasks of paradata analysis analysis
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
The surveyThe survey business R&D activities
yearly , census
target population: enterprises known, or assumed, to be R&D performers
many administrative sources to identify the target population (20000 “potential R&D performers” )(previous Istat R&D surveys, other Istat business surveys with R&D-related questions, ASIA business register, fiscal data, Italian Register of R&D performing institutions,data on national and EU funding to research projects, patent databases, private business reports)
no a cut-off point in term of size, i.e. considered all enterprises (including micro-enterprises, provided that they are employing at least one researcher)
reference year 2008; overall response rate 55%
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Future R&D Future R&D performers performers
(who are just (who are just asked to asked to
report about report about their future their future investment investment
plans)plans)
Potential R&D performersPotential R&D performers
Questionnaire Questionnaire
AActual R&D ctual R&D performers performers (those who have (those who have to provide to provide extensive extensive information on information on the R&D the R&D activities activities undertaken in undertaken in the reference the reference year)year)
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Non-R&D Non-R&D performers (no performers (no data requested)data requested)
R&D R&D personnelpersonnel
Actual R&D performersActual R&D performers
Questionnaire Questionnaire
R&D R&D expendituresexpenditures
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Qualitative Qualitative questions on the questions on the R&D projectsR&D projects
Web survey administrationWeb survey administration
Advanced web questionnaire design
questionnaire mainly based on previous paper questionnaire
innovative software architecture : two physically distinct components
electronic form: delivered via Web on the remote PC of the respondent
checking tool; resident in a Web-server in the Istat’s premises. This is a tool for the identification of the respondent and of the correct sequencing of the questionnaire’s provision.
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Available paradataAvailable paradata
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
•Timing of use:Entry and exit date.Days in which the tool has been accessed.Hours of use during the day.
•Intensity of use:Number and typology of actions (events).Processing time associated to each single action.Number of actions per unit of time (e.g. per day or fraction of day).Outcome of the compilation process.
•Compilation routes:Sequencing in accessing the questions.Changes to previously compiled questions.
•Generation of errors:Number and typology of errors.Questions (or groups of questions) mostly affected by errors.
DefinitionsDefinitions
““eventsevents”:
each time a respondents is interacting with the electronic form (basically, by typing a figure or a word, or even, by scrolling down the form itself).
We can identify each single “event” by its nature and by the time when it took place, as well as by its duration in time (usually, from a fraction to a few seconds).
Events could be either “correct” (according to the rationale behind the structure of the questionnaire) or leading to the generation of “errors”.
The questionnaire is “completed” only when all “errors” are properly fixed, the generation of “errors” will have as a result the need for new events aimed at correcting them. In the process of numbering the errors produced by each respondent, only the first appearance of an error type was taken into consideration, even in the event that the same error type would have been repeated in more than one session.
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
DefinitionsDefinitions
Each event – or a group of events – is associated to an “access”.
““access”:access”: itself to the questionnaire cannot be identified in a straightforward way as “log-ins” and “log-outs” are not recorded in the system.
We know that a respondents is (was) connected to the server because of the activity carried out on the questionnaire.
An access without any activity on the questionnaire will not be taken into consideration. An access has to be qualified in terms of time. Conventionally, all events taking place within one hour (or three, or six hours) could form an “access”.
Only “daily accesses”“daily accesses” are taken into consideration in this study, i.e. an “access” will be equivalent to the set of all events having taken place in a day.
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Results. Results. Business R&D survey 2008 data collection. Paradata indicators: Number of respondents*, daily accesses and errors.
Paradata indicatorsActual R&D performers
Future R&D performers
Non R&D performers
Respondents who have accessed the
questionnaire without completing it
Total
N. of respondents* 2.601 268 1.086 324 4.279 Number of events 614.273 10.942 30.577 5.274 661.066
Average n. events per respondent
236,17 40,83 28,16 16,28 154,49
Number of daily accesses
4.471 371 1.359 391 6.592
Average n. accesses per respondent
1,72 1,38 1,25 1,21 1,54
Average number of events per access
137,39 29,49 22,50 13,49 100,28
Number of errors** 48.291 860 1.852 468 51.471 Average number of errors per respondent
18,57 3,21 1,71 1,44 12,03
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
ResultsResults
Type of errors.Type of errors.
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Frequency of errors by type. Total number of errors=51,533
0
500
1000
1500
2000
2500
3000
3500
4000
E00
1
E01
1
E02
1
E03
1
E04
1
E05
1
E06
1
E07
1
E08
1
E09
1
E10
1
E11
1
E12
1
E13
1
E14
1
E15
1
E16
1
E17
1
E18
1
E19
1
E20
1
E21
1
E22
1
E23
1
E24
1
E25
1
E26
1
E27
1
E28
1
E29
1
E30
1
E31
1
E32
1
E33
1
E34
1
E35
1
E36
1
E37
1
E38
1
Daily accessesDaily accesses
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Frequency of daily accesses by respondent. Actual R&D performersTotal number of accesses=4,478
1
10
100
1000
10000
1 2 3 4 5 6 7 8 9 10
Totale
Somma di Number of enterprises
Number of daily accesses
Final delivery of questionnaireFinal delivery of questionnaire
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Final delivery of questionnaires (Actual R&D performers, 2,200 questionnaires). Observation period: March to October 2010
0
20
40
60
80
100
120
140
05
/03
/20
10
12
/03
/20
10
19
/03
/20
10
26
/03
/20
10
02
/04
/20
10
09
/04
/20
10
16
/04
/20
10
23
/04
/20
10
30
/04
/20
10
07
/05
/20
10
14
/05
/20
10
21
/05
/20
10
28
/05
/20
10
04
/06
/20
10
11
/06
/20
10
18
/06
/20
10
25
/06
/20
10
02
/07
/20
10
09
/07
/20
10
16
/07
/20
10
23
/07
/20
10
30
/07
/20
10
06
/08
/20
10
13
/08
/20
10
20
/08
/20
10
27
/08
/20
10
03
/09
/20
10
10
/09
/20
10
17
/09
/20
10
24
/09
/20
10
01
/10
/20
10
08
/10
/20
10
15
/10
/20
10
22
/10
/20
10
29
/10
/20
10
Actual R&D performers: events accesses and errorsActual R&D performers: events accesses and errors
Using survey data collection as a tool for improving the
survey process
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Number of employees
Number of events for resp.
Number of daily accesses for resp.
Number of errors for resp.
A 1-9 215,7 1,7 18,2
B 10-49 232,2 1,6 17,9
C 50-249 234,3 1,6 18,5
D 250-499 209,9 1,8 17,9
E 500+ 291,0 2,1 21,8
Total 236,2 1,7 18,6
Research in progressResearch in progress
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
deeper analyses on delivering time and on sectoral break-down
more detailed cross checking of the errors and mapping
data processing of the recent R&D survey edition
conclusions on questionnaire improvement
conclusions/suggestions for more efficient follow up strategies
EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss
Suggestions? …..Suggestions? …..
Thank you for your attention!Thank you for your attention!