Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials

Dae Y. Kim, Principal InvestigatorAmy Cassata, Co-Principal InvestigatorJeanne Century, Co-Principal Investigator

IES Grantee Panel on Fidelity of ImplementationSeptember 7, 2011

Instrument Development• NSF-Funded Development Project (2007-2010)

• Developed a framework for measuring the enactment of five reform-based mathematics and science curricula

• Created a suite of instruments based on this framework• Driving Research Questions

• What are the enacted components of reform-based instructional materials in mathematics and science that are common across specific programs?

• How can we rigorously and specifically measure multiple components of implementation?

• How can we collect data on and accumulate knowledge about implementation across multiple programs?

Instrument Development• Reform-based Curricula

• Full Option Science System (FOSS), Science and Technology for Children (STC), Science Companion, Science Education for Public Understanding Program, and Everyday Mathematics

• “Critical Components” • The measurable elements of the intended program model that are

essential to its implementation• May be shared across programs or unique to specific programs• Can be organized into general categories

• Structural: design, procedures, organization, and built-in supports • Instructional: expectations for participants’ actions, behaviors and

interactions as they enact and engage in the intervention (users and recipients)

Framework for Measurement• Process of Critical Component Identification

• Review of written materials to identify explicit and implicit components• Interviews with program developers to articulate the “intended”

program model and critical components• Conversations with program users about critical components• Iterative revision, reconciliation, and modification of critical

components list and framework for organizing them

FOI Framework for Instructional Materials

Measuring Implementation• Suite of instruments measuring the operationalization of

the identified critical components• Teacher Instructional Questionnaire• Teacher Instructional Log• Teacher Instructional Observation Protocol• Teacher Interview Protocol• School Leader Questionnaire• School Leader Interview Protocol• School-Wide Observation Protocol

• Pilot tested in 200 classrooms in CPS (2008) and revised• Field-tested in 265 classrooms in CPS (2009)

Measuring Implementation

Classroom-Level Instruments

Critical Components

Number of Items

Teacher Instructional

Questionnaire

Teacher Instructional Observation

Protocol

Teacher Instructional

Log

Structural Procedural 29 44 39

Structural Educative 4 0 4

Instructional Pedagogical 30 35 31

Instructional Student Engagement 32 32 32

Next Steps: Validation• IES Funding: Goal 5 (Measurement) Goals

• Establish validity and reliability within and across the three classroom-level instruments

• Develop a student questionnaire (grades 3-5) to triangulate measures of student engagement

• Participants• 50 schools across 4 sites (IL, WA, MA)• All teachers grades K-5 (N~1000)• All consented students grades 3-5 (N~4500)• Schools use both Everyday Mathematics and a reform-based

science curriculum

Year 1: Student Questionnaire• 20 to 25-item questionnaire• Literature review for items measuring Student Engagement

critical components• Item development and pilot testing

• Cognitive interviews• Consult with experts

• Field testing• 300 students in grades 3-5• Analyze factorial validity and internal consistency• Instrument revision

Year 2: Data Collection for Validation• All teachers will complete

• Teacher Instructional Questionnaire – Math• Teacher Instructional Questionnaire – Science

• All students (grades 3-5) will complete • Student Questionnaire – Math• Student Questionnaire – Science

• 300 teachers with additional observations and logs• 100 teachers: 2 Math observations and 4 Math logs• 100 teachers: 2 Science observations and 4 Science logs• 100 teachers: 1 Math & 1 Science observation, 4 Math & 4 Science

logs• Student achievement data (grades 3-5)

Year 3: Analysis• Inter-rater agreement (observation)• Internal consistency of items• Factorial structure of items• Differences in scores across different groups/content

areas• Science vs. mathematics• Grade level• Study site

• Cross-instrument consistency • Teacher Instructional Questionnaire vs. Student Questionnaire• Teacher Instructional Observation Protocol vs. Teacher Log

• Predictive validity• Mathematics and science achievement (grades 3-5)

Predicting Student Achievement• To what extent do composite indices on the four

subcategories (SP, SE, IP, ISE) predict student achievement in science and mathematics?

• Planned analysis for predictive validity• 3-level fixed effects HLM model will explore the effect of the FOI

composite indices measured at the teacher (classroom) level on individual students’ science and mathematics achievement (Grades 4-5)

• Prior year’s science or math score will be entered as a covariate• Other variables: student demographics, classroom characteristics• Data for science and mathematics will be analyzed separately,

then pooled• Data for questionnaires, logs, and observations will be analyzed

separately

Summary• The FOI framework clearly and specifically describes the

nature of reform-based mathematics and science instructional materials so these elements can be measured

• The framework has resulted in a set of instruments which are currently being validated

• The framework, and related instruments, have the potential to inform our understanding reform-based STEM curricula and to accumulate knowledge about the elements that make them most effective