WP3 Task3 Fieldwork Monitoring Plans andGoals -...
Transcript of WP3 Task3 Fieldwork Monitoring Plans andGoals -...
WP3 Task3
Fieldwork Monitoring
1
Johanna Bristle
1st DASISH Quantitative WorkshopMannheim, 13 Dec 2012
Fieldwork MonitoringPlans and Goals
Outline
�Description of WP3 Task 3
�Components of a fieldwork monitoring system:
1) Tool (data collection, sample management)
2) Data (paradata, indicators)
3) Monitoring (reports, intervention strategies etc)
2
Description WP3.3
�Objectives:
“To build up and computerize a fieldwork monitoring system
for cross-national surveys which can centrally manage their distributed fieldwork, helping to optimize response rates and data quality” (DOW-283646).
�Tasks and Deliverables:
3
�Tasks and Deliverables:
1. D3.6: Design of a standardized SMS (June 2013)
2. Preparing paradata set (June 2013)D3.7: Report on keystroke analysis (Dec 2014)
3. D3.8: Fieldwork monitoring application (Dec 2014)
Survey Agency
SD
CentERdata
CAPI
SMS
FTP
FTP
Dataflow for Monitoring in SHARE
MEA
Report
FTP
4
Interviewer Laptop
…Interviewer
Laptop
SMSTechnical Exports/Flat Files
KeystrokesSurvey Agency
Country Team
Area Coordinator
MEA
Tool: Standardized SMS
�Goal: Having a transportable, standardized Sample
Management System (SMS)
�Starting Point: SHARE‘s SMS (programmed by CentERdata)
�Steps:�Steps:
1. Adapting SHARE‘s SMS for the ESS and other surveys
• Requirement Needs Survey: Input from surveyagencies/country teams
• Requirements from a coordination point of view
2. Input from other decentralized, cross-national surveys
3. Demo: „How an SMS app might look like“
5
SHARE’s SMS
Tasks:
• Manage fieldwork and sample
• Document contact attempts
• Assign eligibility
• Conduct coverscreen
• Start CAPI-interviews
User:
• Interviewer, installed on laptop
• Need training (at TTT and NTS)
Benefits:
• Data on fieldwork is available during fieldwork
• Fieldwork procedure is comparable across all SHARE-countries
Requirement Needs Survey
�Topics:
�Agency procedures: field force, current system of sample management, training
�Technical requirements: availability of equipment(laptop/tablets/smartphones), technical expertise
�Monitoring: quality checks, procedures�Monitoring: quality checks, procedures
�Perspective of the country teams
�Coordination point of view:
�Standardization vs. decentralized autonomy
�Data collection mode: electronically
7
SMS on a tablet advice
�Draft Demo: SMS app programmed by CentERdata
�http://cdata3.uvt.nl/crs
�One ESS-country already uses an �One ESS-country already uses an app for contact protocols (Spain)
8
For discussion later:
What are your experiences on sample management tools in decentralized surveys?
Data: Paradata set
Type of paradata Concepts/Variables Release
SHARE Keystrokes& time stamps
Call records(from SMS)
Length of interview/module/itemTimes of changing an answer
For each contact attempts/Contact-mode, date, time-result
No
No
9
Neighborhoodinformation(interviewerobservation)
-resultReason for refusal
building type, size of city, stairs toentrancePresence of children, evidence fordisability, intercom
Most
Source: Stoop et al. 2010
Data: Paradata set
Type of paradata Concepts/Variables Release
ESS Time stamps
Call records
(contact protocols)
Length of interview
For each contact attempts/Contact-mode, date, time-resultReason for refusal, age and gender
Partial
Yes
10
Neighborhoodinformation
(interviewerobservation)
Reason for refusal, age and genderfor each refusing person
NUTS, vandalism, littering, buildingtype, physical state of building)
Yes
Monitoring: Components
�SHARE:
�Biweekly monitoring reports based on paradata (send tosurvey agencies, country teams, area coordinators)
�Final compliance profiles
�Quality checks (by core team, country teams and surveyagencies)agencies)
�ESS:
�Country teams monitor and report to assigned core teammember on a (bi-)weekly basis, they report to Sally
�Projection of fieldwork based on completed interviews
�Internal quality checks by core team
11
Survey Agency CentERdata MEA
CAPI
SMS
FTPSD
FTP
FTP
ReportMonitoring filesSMS exports
Statistics
Monitoringreport
Monitoring Output in SHARE
12
Interviewer Laptop
SMSTechnical Exports/Flat Files
KeystrokesSurvey Agency
Country Team
Area Coordinator
MEA
…Interviewer
Laptop
SMS exports
What MEA reports
�MEA gives a report on fieldwork progress every other week
�Separate figures for panel & refreshment sample
�Separate figures for HH & individual level
�Total numbers & ratios:
�Contact attempts�Contact attempts
�Contacts
�Cooperation
�Refusals
�Response/Retention
13
System output
� “Statistics“ tab and export (in SD at survey agency)
� Gives (final) household state of assigned hh / interviewer
� Gives latest individual event codes of detected respondents in assignedhh / interviewer
� Monitoring files provided by Centerdata
14
� Additional info on interviewer performance
� SMS exports (excel sheets)
� Contains all information loaded into / updated by / generated by SMS / SD
� Basis for monitoring reports by MEA
Example 1 (Source: SMS)
60
80
100ATBfBnCHCZDEDKEE
% of HH with at least 1 contact attempt
C O
U N
T R
I E
S
0
20
40
Nov11
Nov26
Dec10
Jan0
7Ja
n21
Feb04
Feb18
Mar
04M
ar18
Apr01
Apr15
Apr29
May
13M
ay27
Jun1
0Ju
n24
EEESFRgFRiHUITNLPTSE
Report 7: Data available to MEA by 24 Jun 2011.
C O
U N
T R
I E
S
Example 2 (Source: keystrokes)
60
80
leng
th in
sec
onds
Time needed for Reading out Item SN001
0
20
40
leng
th in
sec
onds
DE Bn NL FRg DK EE ES CH AT CZ PT IT HUexcludes outside valuesReport 2: Data available to MEA by 22 March 2011.
C O U N T R I E S
Survey Agency CentERdata MEA
CAPI
SMS
SD
FTP
FTP
Report
Questions and Comments
FTP
Interviewer Laptop
SMSTechnical Exports/Flat Files
KeystrokesSurvey Agency
Country Team
Area Coordinator
MEA
…Interviewer
Laptop
17
References
�Stoop, Ineke, Hideko Matsuo, Achim Koch and Jaak Billiet (2010): Paradata in the European Social Survey: Studying Nonresponse and Adjusting for Bias. Section on Survey Research Methods – JSM 2010, p.407-421.
18
Monitoring SHARE
Data Sources
MEA:
• Biweekly reports
• Final report on fieldwork
• Interested in country level
• SMS Files• Monitoring Files
(from CentERdata)• Keystrokes
Country Team:
• Quality checks (surveillance of SA)
Survey Agencies (SA):
• Quality checks on interviewers (mostly
personal contact by phone and email)
• Interested in interviewer level
• Monitoring Files (from CentERdata)
• SMS Files (some)• Keystrokes (some)
Gross sample(drawn/preloaded)
Sample Distributor
SMS client/ Agency
Country teamsMEA
- Monitor fieldwork- Produce response/
retention rates- Release data- Consolidated gross
sample- Preload
- Monitor fieldwork- Payment of agency
SMS client/ CAPI on laptop
Interviewer
CentERdata
SMS/CAPI output- SMS data
- Coverscreen- Fieldwork
- Keystroke data- CAPI data
- Manage contacts- Conduct interviews
- Preload
- Build software- Set up SD with gross
samples- Technical support- Process uploaded data- Data extraction- Translation process
- Manage interviewers- Monitoring fieldwork- billing
SMS data
Household level file
• Panel/refresher HH
• Screening Information
• Coverscreen
• Area Information
Respondent level file
• Demographics
• Eligibility
• Info on CAPI-interview
• Info from Coverscreen
• Contact Information• Participation in previous waves• Event Codes (Intw. Completed,
Refusal, etc.)
• Interviewer Identification
Calculations for Monitoring:• Completed Interviews• Refusal Analysis• All Monitoring Files from
CentERdata
Monitoring Files (from CentERdata)
Fieldwork Indicators for Monitoring at MEA:• % of Contacted Households• Active Interviewers• Progress Rate (for Panel)• Participation Rate at a given point in time• Weekday and Time Slot of Contact
Keystroke data
Calculations for Monitoring at MEA:• Length of interview/module/item• Quality check on reading out items