Post on 20-Jan-2016
Study Overview
National Research Coordinators Meeting
Amsterdam, October 16-19
NRCMeeting
Amsterdam16-19 Oct 2006
IEA studies in the field
• 1971: First IEA Civic Education Study in 10 countries (part of six-subject study)
• 1999/2000: IEA CIVED– Assessment of 14-year-olds in 28
countries– Assessment of upper secondary students
in 16 countries
• 2008/2009: IEA ICCS (more than 40 expressions of interest so far)
NRCMeeting
Amsterdam16-19 Oct 2006
International Study Centre and partner institutions - 1
• Australian Council for Educational Research (ACER), Melbourne/Australia– International Study Centre– Overall Project management– Test Development– Student Background Questionnaire– Scaling and Data analysis
• John Ainley (Project Coordinator)• Wolfram Schulz (Research Director)
NRCMeeting
Amsterdam16-19 Oct 2006
International Study Centre and partner institutions - 2
• National Foundation for Educational Research (NFER), Slough/UK– National Contexts Survey– Student Perceptions Questionnaire– European Regional Module
• David Kerr (Associate Research Director)
NRCMeeting
Amsterdam16-19 Oct 2006
International Study Centre and partner institutions - 3
• Laboratorio di Pedagogia Sperimentale (LPS), Rome/Italy– School Questionnaire– Teacher Questionnaire
• Bruno Losito (Associate Research Director)
NRCMeeting
Amsterdam16-19 Oct 2006
International Study Centre and partner institutions - 4
• IEA Data Processing Center (DPC), Hamburg/Germany– Sampling and Weighting– Data Processing– Scaling and Analysis
• Falk Brese (Coordinator)
• Heiko Sibberns (TEG and PAC representative)
NRCMeeting
Amsterdam16-19 Oct 2006
International Study Centre and partner institutions - 5
• IEA Secretariat, Amsterdam/Netherlands– Translation Verification– Quality Monitoring
• Barbara Malak (Manager Membership Relations)
• Hans Wagemaker (Executive Director)
NRCMeeting
Amsterdam16-19 Oct 2006
Project Management
• Joint Management Committee (JMC)
• Meets frequently (≥ monthly) with consortium staff
• Meets occasionally as a JMC
• Project Advisory Committee (PAC)
• 1st meeting July 2006
• 2nd meeting October 2006
• IEA Secretariat
• IEA Technical Expert Group (TEG)
NRCMeeting
Amsterdam16-19 Oct 2006
Purposes of study
• International comparison of outcomes of civic and citizenship education at start of 21st century
• Provide trend data for comparison with CIVED 1999
• Response to challenge in civic and citizenship education in changed contexts of democracy and participation
NRCMeeting
Amsterdam16-19 Oct 2006
ICCS and CIVED
• Opportunity to review changes
• Extended framework and broadening of scope
• Different approach to teacher survey
• Regional modules
NRCMeeting
Amsterdam16-19 Oct 2006
Research questions directed at exploration of
• Variations in student achievement• Changes since 1999• Extent of interest and dispositions towards
citizenship• Student perceptions of recent threats and
future developments• Relationship of learning outcomes with
curriculum, teaching and school organisation
• Relationship of learning outcomes with student background
NRCMeeting
Amsterdam16-19 Oct 2006
Assessment Framework
• Outlines what will be assessed in ICCS
• Consists of two parts:– Civics and Citizenship framework– Contextual framework
• Current version to be discussed at this meeting
NRCMeeting
Amsterdam16-19 Oct 2006
Instrumentation
• Student instruments comprising:– International (core) cognitive test (60 minutes)– International (core) student background
questions (15 minutes)– International (core) student perceptions on civic-
related issues (30 minutes)– Regional cognitive test module(s) (20-30
minutes) – Regional student questionnaire on civic-related
issues (10-20 minutes)• Teacher questionnaire (30 minutes)• School questionnaire (30 minutes)• National Contexts Survey (on-line)
NRCMeeting
Amsterdam16-19 Oct 2006
Instrument development
• Establishing assessment framework
• Piloting of items in subset of participating countries (Feb-Mar 07)
• Review with experts and NRCs
• Field Trial of preliminary set of instruments (Oct-Nov 07)
• Review with experts and NRCs
• Final selection (Feb-Apr 08)
NRCMeeting
Amsterdam16-19 Oct 2006
General survey design
• Population: Grade 8 (minimum average age of 13.5)
• Two-stage cluster samples in participating countries– PPS sampling of (at least 150) schools– Random selection of one classroom within each
sampled school
• Random selection of teachers within sampled schools (no student-teacher link) teaching grade 8
NRCMeeting
Amsterdam16-19 Oct 2006
Instrument administration
• Test administrators responsible for test and questionnaire sessions
• Student instrument administration:– International cognitive test– International background questionnaire– International perceptions questionnaire– Regional Cognitive test (optional)– Regional perceptions questionnaire
(optional)
NRCMeeting
Amsterdam16-19 Oct 2006
Translation and verification
• Instruments translated from (English) source version into national languages
• National adaptations reviewed in cooperation with international study centre
• Translations verified by language experts
• Final layout review
NRCMeeting
Amsterdam16-19 Oct 2006
Field Operations
• Manuals will be developed to guide operations undertaken by national centres (coordination at school, test administration, coding, data entry)
• School and teacher questionnaires administered through school coordinators
• International control monitors visiting sub-samples of schools
NRCMeeting
Amsterdam16-19 Oct 2006
Data Management and Scaling
• DPC creates international database (cleaning procedures)
• DPC computes sampling weights• DPC provides item statistics• DPC and ACER undertake scaling of data
– IRT scaling model still to be decided (1-PL vs. 3-PL)
– Plausible values for cognitive measures– IRT weighted likelihood scores for some
questionnaire-based indices
NRCMeeting
Amsterdam16-19 Oct 2006
Analysis and Reporting
• (Probably) Jackknife Repeated Replication (JRR) to provide correct standard errors
• Described scales (cognitive and some affective-behavioural)
• Descriptive tables with uni- and bivariate analyses
• Single and multi-level analyses
NRCMeeting
Amsterdam16-19 Oct 2006
Reports
• International Report (June 2010)
• Regional Report(s) (June 2010)
• Technical Report (Nov 2010)– Instrument development– Field Operations– Sampling– Scaling– Reporting
NRCMeeting
Amsterdam16-19 Oct 2006
International Database
• Student, teacher and school databases including:– ID variables– Item responses– Cognitive Scale(s) – Plausible Values– Affective-Behavioural Scales– Weights– JRR Indicators
NRCMeeting
Amsterdam16-19 Oct 2006
Assistance to NRCs
• Item development workshop (this meeting)
• Data Management and Marker Training seminars (FT and MS)
• International Database Seminar (Nov 2010)
NRCMeeting
Amsterdam16-19 Oct 2006
TimelinePlanning and Design May 2006 to October 2006 NRC Meeting 1 October 2006 Assessment Framework May 2006 to January 2007 Pilot January 2007 to May 2007 NRC Meeting 2 June 2007 Field trial July 2007 to January 2008 Main study preparation February 2008 to September 2008 NRC Meeting 3 March 2008 Main study southern hemisphere October 2008 to December 2008 Main study northern hemisphere February 2009 to April 2009 Data compilation and analysis January 2009 to November 2009 NRC Meeting 4 November 2009 Reports and database preparation December 2009 to September 2010
NRCMeeting
Amsterdam16-19 Oct 2006
Questions?