Data Management Seminar, 8-11th July 2008, Hamburg ICCS 2009 Main Survey Field Operations.
-
Upload
asher-anthony -
Category
Documents
-
view
215 -
download
0
Transcript of Data Management Seminar, 8-11th July 2008, Hamburg ICCS 2009 Main Survey Field Operations.
Data Management Seminar, 8-11th July 2008, Hamburg
ICCS 2009 Main SurveyField Operations
Data Management Seminar, 8-11th July 2008, Hamburg
Field Operations - Overview
Main Survey Operations OverviewOn-line Data CollectionSupport MaterialsMain Survey TimelineNext StepsContact Information
Data Management Seminar, 8-11th July 2008, Hamburg
Main Survey Operations Overview
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Verification of
Adaptations
TranslationVerification
Adapt Instruments & DocumentAdaptations
CorrectAdaptations &
TranlsateInstruments
CorrectInstruments &
AssembleBooklets
LayoutVerification
CorrectInstruments
(if necessary)
Printing &Send Out
Instrument Preparation
Data Management Seminar, 8-11th July 2008, Hamburg
NRCISC/IEA Sec./DPCtime
Time planning before administration
Layout verification
Implement changes &assemble booklets
Translation verification
Adapt instruments
2 weeks
1 week
1 week
2 weeks
4 weeks
Start data collection15
weeks
Instruments send out
Printing & send out
Translate instruments
Verification of adaptations
1 week
4 weeks
Data Management Seminar, 8-11th July 2008, Hamburg
due dateNRC/ISC/IEAtime
Time planning before administration - SH
Layout verification
Implement changes &assemble instruments
Translation verification
Adapt & translate instruments
1 week
1 week
2 weeks
1 week
Instruments send out July 11, 2008
August 22, 2008September 19,
2008September 26,
2008October 10,
2008
October 24, 2008Start data collection
Printing & send out2 weeksOctober 24,
2008
Adapt instruments
Verification of adaptations July 25, 2008
July 18, 2008
4 weeks
4 weeks
Data Management Seminar, 8-11th July 2008, Hamburg
Meanwhile…
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Contact Schoolsand get
Within-SchoolInformation
Perform Within-School
Sampling
Prepare List of Schools
and send it to DPC
Recruit and Train Test Administrators
DPC provides list of sampled
Schools
Recruit and Train Scorers
Translate School Coordinator Manual
Translate Test Administrator Manual
Recruit and Train School Coordinators
Sampling & Preparing Survey Admin.
Data Management Seminar, 8-11th July 2008, Hamburg
School Contact
If not yet done, send sampling frame (list of schools) to the IEA DPCSampling of schools has been or will be done by the IEA DPCList of sampled schools has been or will be provided by the IEA DPCContact schools, ensure participationIdentify liaison person (School Coordinator)Provide School Coordinator ManualRecord status for each sampled school in WinW3S
Data Management Seminar, 8-11th July 2008, Hamburg
School Contact
Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible, also combining the two to only one document)
Data Management Seminar, 8-11th July 2008, Hamburg
Within-School Sampling (Students)
Two levels: students and teachersIdentification of classes on Class Listing Forms– Class name– Number of students
Random within-school sample of classesIdentification of students within sampled classes on Student Listing FormsSend list of sampled students to School Coordinator on Student Tracking FormWinW3S serves as survey administration tool
Data Management Seminar, 8-11th July 2008, Hamburg
Within-School Sampling (Teachers)
Identification of eligible target grade teachers on Teacher Listing FormsWithin-school random sample of target grade teachersSend Teacher Tracking Form to School Coordinator
Data Management Seminar, 8-11th July 2008, Hamburg
Within-School Sampling
WinW3S will support NRCs and DMs with– Storing school contact information– Listing of classes / teachers / students– Sampling of classes / teachers– Assignment of booklets– Survey administration– Tracking of participation
Will be subject today and tomorrow
Data Management Seminar, 8-11th July 2008, Hamburg
Assigning Questionnaires
School QuestionnaireTeacher– Every teacher will receive one questionnaire
Student– Achievement Booklet (one of seven rotated forms)– Questionnaire (one form, no rotated forms in main
survey)– Regional Module Questionnaire
WinW3S will assign forms and provide labels for questionnaires
Data Management Seminar, 8-11th July 2008, Hamburg
NRCSchoolstime
Contacts schools
Decision to participate
Sends listing forms
List students/teachers
Samples teachers
Sends tracking forms
Sends questionnaires
Start data collection
Name School Coordinator
Samples classes
List classes/providenumber of teachers
Time Planning before Administration
Data Management Seminar, 8-11th July 2008, Hamburg
School ContactInstrumentstime
Contact schools
Send listing forms
Sample teachers
Send tracking forms
Send questionnaires
Implement changes & assemble
booklets
Adapt & translate instruments
Printing
Data collection
Sample classes
Time Planning before Administration
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Monitor Test Administrators
Ad hoc Support
Arrange for Material Receipt
Train Scorers
Train Data Entry Staff
Survey Administration
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Organize Material for (direct) Data Entry
Organize Material for Scoring and Reliability Scoring
Check if Information and Material is complete
Get back to School
if necessary
Enter Participation Information
Receiving Material
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Adapt InternationalCodebooks to
National Questionnaires
Use NationalAdaptation Forms
Use adaptedCodebooks for
Data Entry in WinDEM
Codebook Adaptation
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Conduct and Monitor Scoring
Conduct and Monitor
Reliability Scoring
Organize Material forData Entry
Conduct and Monitor
Reliability Scoring
Conduct and Monitor Scoring
Retrain orDiscuss items
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Scoring
All open ended items have to be scored according to scoring guideIdeally at the beginning of the scoring process, the first incoming booklets should be scored for reliability, so that measures can be undertaken to improve reliability of scoring (e.g., discuss and retrain problematic items)Reliability scoring required for open ended items for ~700 booklets (100 booklets per rotated form) Occupations of parents/guardians have to be scored according to ISCO 88
Has been subject to Marker Training yesterday and today
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Complete Participation Information
in WinW3S
Verify DataEnterData
in WinDEM
Data Entry
EnterDouble Punching Data
in WinDEM
Data Management Seminar, 8-11th July 2008, Hamburg
Data Entry
Data entry software (WinDEM) and international codebooks for the MS are provided by the DPC on August 18, 2008National staff should be trained on data entry, even if all or parts are being outsourcedComputer workstations need to be set upNational Centers enter data and double punching data in WinDEMBefore send out, National Centers need to verify data from paper questionnaires using verification checks provided in WinDEM
Will be subject tomorrow through Thursday
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
Set of Final Instruments
National Adaptation Forms(NAF) version IV
Exported WinW3S database
National Codebooks
Verified Data sets
Reliability Scoring Data
Data Submission
Documentation Data
Tracking Forms Double Punching Data
Data Management Seminar, 8-11th July 2008, Hamburg
InstrumentPreparation
Sampling and Preparing
Survey Administration
Data Entryand Verification
SURVEY
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Sendout
Scoring
Data Management Seminar, 8-11th July 2008, Hamburg
On-line Data Collection
Data Management Seminar, 8-11th July 2008, Hamburg
On-line Data Collection (ODC)Used for all questionnaires that National Centers should answer– National Context Questionnaire– Survey Activities Questionnaire
International Option for– School Questionnaire– Teacher Questionnaire
Analysis of split half design during field trial showed that the use of the on-line mode in general is feasibleAll nine countries participating in ODC option during the FT received individual feedbackOnly countries that tried out ODC in FT can use ODC option for MS
Data Management Seminar, 8-11th July 2008, Hamburg
On-line Data Collection in MS
True mixed mode designGoal:– Reduce response burden by offering respondents
preferred mode– Save time on data entry and data processing– Improve quality of data
Teachers and school principals should be assigned to the preferred mode– Knowlegde of NRC about feasibility/preferences– School coordinator could contactS school principal
and teachers
Ensure fall-back to paper questionnaire
Data Management Seminar, 8-11th July 2008, Hamburg
Codebook Adaptation
InstrumentPreparation
Sampling and PreparingAdministration
Data Entry and
Verification
Adm
inis
trati
on
Rece
ivin
g M
ate
rial
Data
SendoutCodebook Adaptation
Prepare
ODC Monit
or
OD
C
Data Management Seminar, 8-11th July 2008, Hamburg
School contactInstrumentstime
Time planning before administration ODC
Contact schools
Sample students / teachers
Send tracking forms
Send questionnaires
Implement changes & assemble
booklets
Adapt & translate instruments
Printing
Data collection
PrintingPrepareODC
Send quest./cover letters
Send listing forms
Sample classes
Data Management Seminar, 8-11th July 2008, Hamburg
On-line Data Collection (ODC)
Based on concept of cultures, one setup per language and countryParticipants receive “cover letter”– Label with login information provided by WinW3S– Culture specific URL– Personal username and password– Sent to school coordinator for distribution
Same administration period as paper questionnaires
Data Management Seminar, 8-11th July 2008, Hamburg
On-line Data Collection (ODC)
NRCs can monitor completion status of on-line questionnairesIf necessary, NRCs instruct School Coordinators to remind teachers/school principals to complete the questionnaireNo data entry at National Center needed
Data Management Seminar, 8-11th July 2008, Hamburg
Support Material
Data Management Seminar, 8-11th July 2008, Hamburg
Support Material
Survey Operations Procedures Manual (MS)– Unit 1: School contact and material preparation– Unit 2: Within school sampling and test
administrationSchool Coordinator ManualTest Administrator ManualNational Quality Control Monitor ManualWithin School Sampling Software (WinW3S)
Data Management Seminar, 8-11th July 2008, Hamburg
Support Material
Survey Operations Procedures Manual (MS)– Unit 3: On-line data collection, scoring and data entry
IEA SurveySystem DesignerScoring GuideCodebooksWindows Data Entry Manager (WinDEM)
Distribution of all materials and software packages through secure section of study website at http://iccs.acer.edu.au
Data Management Seminar, 8-11th July 2008, Hamburg
Main Survey Timeline
Data Management Seminar, 8-11th July 2008, Hamburg
Main Survey Timeline - SH
- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires
October 10, 2008
Submit assembled instruments and NAF version III for layout verification to ACER
September 26, 2008
Receive verified instrumentSeptember 19, 2008
Submit translated questionnaires and NAF version II for international translation verification to IEA Sec.
August 22, 2008
Receive feedback on national adaptationsJuly 25, 2008
Submit national adaptations in NAF version I to ISCJuly 18, 2008
Receive instruments & NAFJuly 11, 2008
Data Management Seminar, 8-11th July 2008, Hamburg
Main Survey Timeline - SH
Final deadline for submitting data to DPCMarch 27, 2009
- Scoring of open-ended items and occupations- Data entry and verification
November –March, 2009
Proposed MS administration period (subject to country conditions)
October – December, 2008
Submit on-line questionnaires definition files for verification to DPC
October 17, 2008
Give feedback about MS procedures using theon-line Survey Activities Questionnaire
April, 2009
Data Management Seminar, 8-11th July 2008, Hamburg
Main Survey Timeline – NH
- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires
February, 2008
Submit assembled instruments and NAF version III for layout verification to ACER
February, 2008
Translation verificationJanuary, 2008
Translate questionnaires and prepare NAF version II for international translation verification at IEA Sec.
December, 2008
Receive feedback on national adaptationsNovember, 2008
Submit national adaptations in NAF version I to ISCNovember, 2008
Receive instruments & NAFJuly 11, 2008
Data Management Seminar, 8-11th July 2008, Hamburg
Main Survey Timeline - NH
Final deadline for submitting data to DPCAugust 28, 2009
- Scoring of open-ended items and occupations- Data entry and verification
April –August, 2009
Proposed MS administration period (subject to country conditions)
March – May, 2009
Submit on-line questionnaires definition files for verification to DPC
February, 2008
Give feedback about MS procedures using theon-line Survey Activities Questionnaire
September, 2009
Data Management Seminar, 8-11th July 2008, Hamburg
Next Steps
Data Management Seminar, 8-11th July 2008, Hamburg
Next Steps
National centers should plan and prepare for– School contact– Adaptation & translation of instruments– Test administrators– Scorers (CCE items and occupations)– Data management (data entry and verification)
Be prepared to receive Main Survey instruments on July 11, 2008
Data Management Seminar, 8-11th July 2008, Hamburg
Contact Information
Data Management Seminar, 8-11th July 2008, Hamburg
Contact Information
Field Operations / Data Management:
IEA Data Processing and Research Center
Falk Brese Michael Jung
Data Management Seminar, 8-11th July 2008, Hamburg
Thank you for your attention!