Study Overview National Research Coordinators Meeting Amsterdam, October 16-19.

Study Overview

National Research Coordinators Meeting

Amsterdam, October 16-19

NRCMeeting

Amsterdam16-19 Oct 2006

IEA studies in the field

• 1971: First IEA Civic Education Study in 10 countries (part of six-subject study)

• 1999/2000: IEA CIVED– Assessment of 14-year-olds in 28

countries– Assessment of upper secondary students

in 16 countries

• 2008/2009: IEA ICCS (more than 40 expressions of interest so far)

NRCMeeting


International Study Centre and partner institutions - 1

• Australian Council for Educational Research (ACER), Melbourne/Australia– International Study Centre– Overall Project management– Test Development– Student Background Questionnaire– Scaling and Data analysis

• John Ainley (Project Coordinator)• Wolfram Schulz (Research Director)

NRCMeeting



• National Foundation for Educational Research (NFER), Slough/UK– National Contexts Survey– Student Perceptions Questionnaire– European Regional Module

• David Kerr (Associate Research Director)

NRCMeeting



• Laboratorio di Pedagogia Sperimentale (LPS), Rome/Italy– School Questionnaire– Teacher Questionnaire

• Bruno Losito (Associate Research Director)

NRCMeeting



• IEA Data Processing Center (DPC), Hamburg/Germany– Sampling and Weighting– Data Processing– Scaling and Analysis

• Falk Brese (Coordinator)

• Heiko Sibberns (TEG and PAC representative)

NRCMeeting



• IEA Secretariat, Amsterdam/Netherlands– Translation Verification– Quality Monitoring

• Barbara Malak (Manager Membership Relations)

• Hans Wagemaker (Executive Director)

NRCMeeting


Project Management

• Joint Management Committee (JMC)

• Meets frequently (≥ monthly) with consortium staff

• Meets occasionally as a JMC

• Project Advisory Committee (PAC)

• 1st meeting July 2006

• 2nd meeting October 2006

• IEA Secretariat

• IEA Technical Expert Group (TEG)

NRCMeeting


Purposes of study

• International comparison of outcomes of civic and citizenship education at start of 21st century

• Provide trend data for comparison with CIVED 1999

• Response to challenge in civic and citizenship education in changed contexts of democracy and participation

NRCMeeting


ICCS and CIVED

• Opportunity to review changes

• Extended framework and broadening of scope

• Different approach to teacher survey

• Regional modules

NRCMeeting


Research questions directed at exploration of

• Variations in student achievement• Changes since 1999• Extent of interest and dispositions towards

citizenship• Student perceptions of recent threats and

future developments• Relationship of learning outcomes with

curriculum, teaching and school organisation

• Relationship of learning outcomes with student background

NRCMeeting


Assessment Framework

• Outlines what will be assessed in ICCS

• Consists of two parts:– Civics and Citizenship framework– Contextual framework

• Current version to be discussed at this meeting

NRCMeeting


Instrumentation

• Student instruments comprising:– International (core) cognitive test (60 minutes)– International (core) student background

questions (15 minutes)– International (core) student perceptions on civic-

related issues (30 minutes)– Regional cognitive test module(s) (20-30

minutes) – Regional student questionnaire on civic-related

issues (10-20 minutes)• Teacher questionnaire (30 minutes)• School questionnaire (30 minutes)• National Contexts Survey (on-line)

NRCMeeting


Instrument development

• Establishing assessment framework

• Piloting of items in subset of participating countries (Feb-Mar 07)

• Review with experts and NRCs

• Field Trial of preliminary set of instruments (Oct-Nov 07)

• Review with experts and NRCs

• Final selection (Feb-Apr 08)

NRCMeeting


General survey design

• Population: Grade 8 (minimum average age of 13.5)

• Two-stage cluster samples in participating countries– PPS sampling of (at least 150) schools– Random selection of one classroom within each

sampled school

• Random selection of teachers within sampled schools (no student-teacher link) teaching grade 8

NRCMeeting


Instrument administration

• Test administrators responsible for test and questionnaire sessions

• Student instrument administration:– International cognitive test– International background questionnaire– International perceptions questionnaire– Regional Cognitive test (optional)– Regional perceptions questionnaire

(optional)

NRCMeeting


Translation and verification

• Instruments translated from (English) source version into national languages

• National adaptations reviewed in cooperation with international study centre

• Translations verified by language experts

• Final layout review

NRCMeeting


Field Operations

• Manuals will be developed to guide operations undertaken by national centres (coordination at school, test administration, coding, data entry)

• School and teacher questionnaires administered through school coordinators

• International control monitors visiting sub-samples of schools

NRCMeeting


Data Management and Scaling

• DPC creates international database (cleaning procedures)

• DPC computes sampling weights• DPC provides item statistics• DPC and ACER undertake scaling of data

– IRT scaling model still to be decided (1-PL vs. 3-PL)

– Plausible values for cognitive measures– IRT weighted likelihood scores for some

questionnaire-based indices

NRCMeeting


Analysis and Reporting

• (Probably) Jackknife Repeated Replication (JRR) to provide correct standard errors

• Described scales (cognitive and some affective-behavioural)

• Descriptive tables with uni- and bivariate analyses

• Single and multi-level analyses

NRCMeeting


Reports

• International Report (June 2010)

• Regional Report(s) (June 2010)

• Technical Report (Nov 2010)– Instrument development– Field Operations– Sampling– Scaling– Reporting

NRCMeeting


International Database

• Student, teacher and school databases including:– ID variables– Item responses– Cognitive Scale(s) – Plausible Values– Affective-Behavioural Scales– Weights– JRR Indicators

NRCMeeting


Assistance to NRCs

• Item development workshop (this meeting)

• Data Management and Marker Training seminars (FT and MS)

• International Database Seminar (Nov 2010)

NRCMeeting


TimelinePlanning and Design May 2006 to October 2006 NRC Meeting 1 October 2006 Assessment Framework May 2006 to January 2007 Pilot January 2007 to May 2007 NRC Meeting 2 June 2007 Field trial July 2007 to January 2008 Main study preparation February 2008 to September 2008 NRC Meeting 3 March 2008 Main study southern hemisphere October 2008 to December 2008 Main study northern hemisphere February 2009 to April 2009 Data compilation and analysis January 2009 to November 2009 NRC Meeting 4 November 2009 Reports and database preparation December 2009 to September 2010

NRCMeeting


Questions?

Study Overview National Research Coordinators Meeting Amsterdam, October 16-19.

Documents

Transcript of Study Overview National Research Coordinators Meeting Amsterdam, October 16-19.