Post on 16-Jan-2016
3rd NRC Meeting, 9-12 June 2008, Windsor
ICCS 2009 Main Survey
Field Operations
2 3rd NRC Meeting, 9-12 June 2008, Windsor
Field Operations - Overview
National Context Questionnaire
Main Survey Operations Overview
On-line Data Collection
Support Materials
Main Survey Timeline
Next Steps
3 3rd NRC Meeting, 9-12 June 2008, Windsor
National Context Questionnaire
Stage 2 of National Contexts Survey will start in March 2009 and will continue until October/November 2009
Survey will be conducted on-line (similar to stage 1)
NRCs will only need to update information from stage 1
NOTE: There are separate logins (username and password) for the sources available on-line:
– ICCS study homepage
– WebFTP server (2-stage login)
– National Context Questionnaire
– Survey Activities Questionnaire
3rd NRC Meeting, 9-12 June 2008, Windsor
Main Survey Operations Overview
5 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
6 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
7 3rd NRC Meeting, 9-12 June 2008, Windsor
Verification ofAdaptations
TranslationVerification
Adapt Instruments & DocumentAdaptations
CorrectAdaptations &
TranlsateInstruments
CorrectInstruments &
AssembleBooklets
LayoutVerification
CorrectInstruments
(if necessary)
Printing &Send Out
8 3rd NRC Meeting, 9-12 June 2008, Windsor
NRCISC/IEA Sec./DPCtime
Time planning before administration
Layout verification
Implement changes &assemble booklets
Translation verification
Adapt instruments
2 weeks
1 week
1 week
3 weeks
4 weeks
Start data collection16
weeks
Instruments send out
Printing & send out
Translate instruments
Verification of adaptations
1 week
4 weeks
9 3rd NRC Meeting, 9-12 June 2008, Windsor
due dateNRC/ISC/IEAtime
Time planning before administration - SH
Layout verification
Implement changes &assemble instruments
Translation verification
Adapt & translate instruments
1 week
1 week
2 weeks
1 week
Instruments send out July 11, 2008
August 22, 2008
September 19, 2008
September 26, 2008
October 10, 2008
October 31, 2008Start data collection
Printing & send out3 weeks October 31, 2008
Adapt instruments
Verification of adaptations July 25, 2008
July 18, 2008
4 weeks
4 weeks
3rd NRC Meeting, 9-12 June 2008, Windsor
Meanwhile…
11 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
12 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
13 3rd NRC Meeting, 9-12 June 2008, Windsor
Contact Schoolsand get
within SchoolInformation
Perform Within-School
Sampling
Prepare List of Schools
and send it to DPC
Recruit and Train Test Administrators
DPC provides list of sampled
Schools
Recruit and Train Scorers
Translate School Coordinator Manual
Translate Test Administrator Manual
14 3rd NRC Meeting, 9-12 June 2008, Windsor
School Contact
If not yet done, send sampling frame (list of schools) to the IEA DPC
Sampling of schools has been or will be done by the IEA DPC
List of sampled schools has been or will be provided by the IEA DPC
Contact schools, ensure participation
Identify liaison person (School Coordinator)
Provide School Coordinator Manual
Record status for each sampled school in WinW3S
Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible, also combining the two to only one document)
15 3rd NRC Meeting, 9-12 June 2008, Windsor
Within-School Sampling (Students)
Two levels: students and teachers
Identification of classes on Class Listing Forms
– Class name
– Number of students
Random within-school sample of classes
Identification of students within sampled classes on Student Listing Forms
Send list of sampled students to School Coordinator on Student Tracking Form
WinW3S serves as survey administration tool
16 3rd NRC Meeting, 9-12 June 2008, Windsor
Within-School Sampling (Teachers)
Identification of eligible target grade teachers on Teacher Listing Forms
Within-school random sample of target grade teachers
Send Teacher Tracking Form to School Coordinator
17 3rd NRC Meeting, 9-12 June 2008, Windsor
Within-School Sampling
WinW3S will support NRCs with
– Storing school contact information
– Listing of classes / teachers / students
– Sampling of classes / teachers
– Assignment of booklets
– Survey administration
– Tracking of participation
Will be subject to Data Management Seminar on July 8-11, 2008 in Hamburg
18 3rd NRC Meeting, 9-12 June 2008, Windsor
Assigning Questionnaires
School Questionnaire
Teacher– Every teacher will receive one questionnaire
Student– Achievement Booklet (one of seven rotated forms)– Questionnaire (one form, no rotated forms in main survey)– Regional Module Questionnaire
WinW3S will assign forms and provide labels for questionnaires
19 3rd NRC Meeting, 9-12 June 2008, Windsor
NRCSchools
Time Planning before Administration
time
Contacts schools
Decision to participate
Sends listing forms
List students/teachers
Samples teachers
Sends tracking forms
Sends questionnaires
Start data collection
Name School Coordinator
Samples classes
List classes/providenumber of teachers
20 3rd NRC Meeting, 9-12 June 2008, Windsor
School ContactInstrumentstime
Time Planning before Administration
Contact schools
Send listing forms
Sample teachers
Send tracking forms
Send questionnaires
Implement changes & assemble
booklets
Adapt & translate instruments
Printing
Data collection
Sample classes
21 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
22 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
23 3rd NRC Meeting, 9-12 June 2008, Windsor
Monitor Test Administrators
Ad hoc Support
Arrange for Material Receipt
Train Scorers
Train Data Entry Staff
24 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
25 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
26 3rd NRC Meeting, 9-12 June 2008, Windsor
Organize Material for (direct) Data Entry
Organize Material for Scoring and Reliability Scoring
Check if Information and Material is complete
Get back to School
if necessary
Enter Participation Information
27 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
28 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
29 3rd NRC Meeting, 9-12 June 2008, Windsor
Adapt InternationalCodebooks to
National Questionnaires
Use NationalAdaptation Forms
Use adaptedCodebooks for
Data Entry in WinDEM
30 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
31 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
32 3rd NRC Meeting, 9-12 June 2008, Windsor
Conduct and Monitor Scoring
Conduct and Monitor
Reliability Scoring
Organize Material forData Entry
33 3rd NRC Meeting, 9-12 June 2008, Windsor
Scoring
All open ended items have to be scored according to scoring guide
Reliability scoring for open ended items for ~700 booklets (100 booklets per rotated form), ideally at the beginning of the scoring process, so that measures can be undertaken to improve reliability of scoring
Occupations of parents/guardians have to be scored according to ISCO 88
Subject to Marker Training on July 7-8, 2008, in Hamburg
34 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
35 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
36 3rd NRC Meeting, 9-12 June 2008, Windsor
Complete Participation Information
Verify Data
Enter Data
37 3rd NRC Meeting, 9-12 June 2008, Windsor
Data Entry
Data entry software (WinDEM) and international codebooks for the MS are provided by the DPC on August 18, 2008
National staff should be trained on data entry, even if all or parts are being outsourced
Computer workstations need to be set up
National Centers enter and verify data from paper questionnaires
38 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
39 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
40 3rd NRC Meeting, 9-12 June 2008, Windsor
Set of Final Instruments
National Codebooks and Documentation (NAF v. IV)
VerifiedData sets
TrackingForms
Reliability Scoring Data
41 3rd NRC Meeting, 9-12 June 2008, Windsor
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sen
dout
Scoring
3rd NRC Meeting, 9-12 June 2008, Windsor
On-line Data Collection
43 3rd NRC Meeting, 9-12 June 2008, Windsor
On-line Data Collection (ODC)
Used for all questionnaires that National Centers should answer
– National Context Questionnaire
– Survey Activities Questionnaire
International Option for
– School Questionnaire
– Teacher Questionnaire
Analysis of split half design during field trial showed that the use of the on-line mode in general is feasible
All nine countries participating in ODC option during the FT received individual feedback
Only countries that tried out ODC in FT can use ODC option for MS
44 3rd NRC Meeting, 9-12 June 2008, Windsor
On-line Data Collection in MS
True mixed mode design
Goal:
– Reduce response burden by offering respondents preferred mode
– Save time on data entry and data processing
– Improve quality of data
Teachers and school principals should be assigned to the preferred mode
– Knowlegde of NRC about feasibility/preferences
– School coordinator could contactS school principal and teachers
Ensure fall-back to paper questionnaire
45 3rd NRC Meeting, 9-12 June 2008, Windsor
Codebook Adaptation
InstrumentPreparation
Sampling and PreparingAdministration
Data Entry and
Verification
Ad
min
istr
ati
on
Rece
ivin
g M
ate
rial
Data
Sen
doutCodebook Adaptation
Prepare ODC
Monit
or
OD
C
46 3rd NRC Meeting, 9-12 June 2008, Windsor
School contactInstrumentstime
Time planning before administration ODC
Contact schools
Sample students / teachers
Send tracking forms
Send questionnaires
Implement changes & assemble
booklets
Adapt & translate instruments
Printing
Data collection
PrintingPrepareODC
Send quest./cover letters
Send listing forms
Sample classes
47 3rd NRC Meeting, 9-12 June 2008, Windsor
On-line Data Collection (ODC)
Based on concept of cultures, one setup per language and country
Participants receive “cover letter”
– Label with login information provided by WinW3S
– Culture specific URL
– Personal username and password
– Sent to school coordinator for distribution
Same administration period as paper questionnaires
NRCs can monitor completion status of on-line questionnaires
NEW: Status of on-line respondents can be exported to WinW3S
If necessary, NRCs instruct School Coordinators to remind teachers/school principals to complete the questionnaire
No data entry at National Center needed
3rd NRC Meeting, 9-12 June 2008, Windsor
Support Material
49 3rd NRC Meeting, 9-12 June 2008, Windsor
Support Material
Survey Operations Procedures Manual (MS)– Unit 1: School contact and material preparation– Unit 2: Within school sampling and test administration
School Coordinator ManualTest Administrator ManualNational Quality Control Monitor ManualWithin School Sampling Software (WinW3S)
50 3rd NRC Meeting, 9-12 June 2008, Windsor
Support Material
Survey Operations Procedures Manual (MS)– Unit 3: On-line data collection, scoring and data entry
IEA SurveySystem DesignerScoring GuideCodebooksWindows Data Entry Manager (WinDEM)
Distribution of all materials and software packages through secure section of study website at http://iccs.acer.edu.au
3rd NRC Meeting, 9-12 June 2008, Windsor
Main Survey Timeline
52 3rd NRC Meeting, 9-12 June 2008, Windsor
Main Survey Timeline - SH
- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires
October 10, 2008
Submit assembled instruments and NAF version III for layout verification to ACER
September 26, 2008
Receive verified instrumentSeptember 19, 2008
Submit translated questionnaires and NAF version II for international translation verification to IEA Sec.
August 22, 2008
Receive feedback on national adaptationsJuly 25, 2008
Submit national adaptations in NAF version I to ISCJuly 18, 2008
Receive instruments & NAFJuly 11, 2008
53 3rd NRC Meeting, 9-12 June 2008, Windsor
Main Survey Timeline - SH
Final deadline for submitting data to DPCMarch 27, 2009
- Scoring of open-ended items and occupations- Data entry and verification
November –March, 2009
Proposed MS administration period (subject to country conditions)
October – December, 2008
Submit on-line questionnaires definition files for verification to DPC
October 17, 2008
Give feedback about FT procedures using theon-line Survey Activities Questionnaire
April, 2009
54 3rd NRC Meeting, 9-12 June 2008, Windsor
Main Survey Timeline – NH
- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires
February, 2008
Submit assembled instruments and NAF version III for layout verification to ACER
February, 2008
Translation verificationJanuary, 2008
Translate questionnaires and prepare NAF version II for international translation verification at IEA Sec.
December, 2008
Receive feedback on national adaptationsNovember, 2008
Submit national adaptations in NAF version I to ISCNovember, 2008
Receive instruments & NAFJuly 11, 2008
55 3rd NRC Meeting, 9-12 June 2008, Windsor
Main Survey Timeline - NH
Final deadline for submitting data to DPCAugust 28, 2009
- Scoring of open-ended items and occupations- Data entry and verification
April –August, 2009
Proposed MS administration period (subject to country conditions)
March – May, 2009
Submit on-line questionnaires definition files for verification to DPC
February, 2008
Give feedback about FT procedures using theon-line Survey Activities Questionnaire
September, 2009
3rd NRC Meeting, 9-12 June 2008, Windsor
Next Steps
57 3rd NRC Meeting, 9-12 June 2008, Windsor
Next Steps
Prepare team at national center and plan for
– School contact
– Adaptation & translation of instruments
– Test administrators
– Scorers (CCE items and occupations)
– Data management (data entry and verification)
Be prepared to receive Main Survey instruments on July 11, 2008
58 3rd NRC Meeting, 9-12 June 2008, Windsor
Next Steps
Marker Training on July 7-8, 2008 in Hamburg
Data Management Seminar on July 8-11, 2008 in Hamburg
3rd NRC Meeting, 9-12 June 2008, Windsor
Contact Information
60 3rd NRC Meeting, 9-12 June 2008, Windsor
Contact Information
Field Operations / Data Management:
IEA Data Processing and Research Center
Falk Brese Michael Jung
iccs@iea-dpc.de
3rd NRC Meeting, 9-12 June 2008, Windsor
Thank you for your attention!