MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09...
-
Upload
madeleine-norton -
Category
Documents
-
view
216 -
download
0
Transcript of MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09...
MSP Program EvaluationMSP Program Evaluation
Carol L. Fletcher, Ph.D.TRC Project Director
Meetings1/27/09 and 2/5/09
Carol L. Fletcher, Ph.D.TRC Project Director
Meetings1/27/09 and 2/5/09
TRC CriteriaTRC Criteria
• How has your Regional Collaborative impacted• Teacher Content Knowledge• Classroom Practice• Student Achievement
• How has your Regional Collaborative impacted• Teacher Content Knowledge• Classroom Practice• Student Achievement
Guskey ModelGuskey Model
• Based on mediators of relationship between professional development and improvements in student learning• i.e. administrator knowledge and
practice, school culture, teacher knowledge and practice, parent knowledge and practice
• Based on mediators of relationship between professional development and improvements in student learning• i.e. administrator knowledge and
practice, school culture, teacher knowledge and practice, parent knowledge and practice
Five Critical Levels of PD Evaluation
Five Critical Levels of PD Evaluation
• Inform planning, formative and summative evaluations
• Helps to identify the “what” and the “why” evaluation
• Hierarchically arranged from simple to complex
• Success at one level is necessary for success at following levels
• Inform planning, formative and summative evaluations
• Helps to identify the “what” and the “why” evaluation
• Hierarchically arranged from simple to complex
• Success at one level is necessary for success at following levels
Level 1: Participants’ ReactionsLevel 1: Participants’ Reactions
• Did they like it?• Will it be useful• Was the leader
knowledgeable and helpful?
• Where the facilities comfortable?
• Did they like it?• Will it be useful• Was the leader
knowledgeable and helpful?
• Where the facilities comfortable?
• Questionnaires• Interviews• Participant
journals
• Questionnaires• Interviews• Participant
journals
What questions are addressed? How will information be gathered?
Level 2: Participants’ LearningLevel 2: Participants’ Learning
• Did participants acquire the targeted knowledge and skills?
• Did participants acquire the targeted knowledge and skills?
• Questionnaires• Interviews• Participant
journals/portfolios• Demonstrations
• Questionnaires• Interviews• Participant
journals/portfolios• Demonstrations
What questions are addressed? How will information be gathered?
Level 3: Organization Support and Change
Level 3: Organization Support and Change
• Did it affect organizational climate, procedures, or policies
• Were sufficient resources made available?
• Were successes recognized & shared?
• Did it affect organizational climate, procedures, or policies
• Were sufficient resources made available?
• Were successes recognized & shared?
• Questionnaires• Structured
Interviews• Participant
journals/portfolios• Focus groups• District/school
records
• Questionnaires• Structured
Interviews• Participant
journals/portfolios• Focus groups• District/school
records
What questions are addressed? How will information be gathered?
Level 4: Participants’ use of new knowledge and skills
Level 4: Participants’ use of new knowledge and skills
• Did participants effectively apply the new knowledge and skills?
• Did participants effectively apply the new knowledge and skills?
• Questionnaires• Structured
Interviews• Participant
journals/portfolios/ reflections
• Direct observation
• Videotapes
• Questionnaires• Structured
Interviews• Participant
journals/portfolios/ reflections
• Direct observation
• Videotapes
What questions are addressed? How will information be gathered?
Level 5: Student learning outcomesLevel 5: Student learning outcomes
• Did it affect student performance or achievement?
• Are students more engaged learners?
• Do students select more rigorous coursework?
• Did it affect student attitudes toward science or math courses/careers?
• Did it affect student performance or achievement?
• Are students more engaged learners?
• Do students select more rigorous coursework?
• Did it affect student attitudes toward science or math courses/careers?
• Questionnaires• School/student
records• Structured
Interviews• Participant
journals/portfolios/ reflections
• Direct observation• Videotapes
• Questionnaires• School/student
records• Structured
Interviews• Participant
journals/portfolios/ reflections
• Direct observation• Videotapes
What questions are addressed? How will information be gathered?
NSF Developed ToolsNSF Developed Tools
• Horizon Research• Assessing Teacher Learning About
Science Teaching (ATLAST)• www.horizon-research.com/atlast/• Instruments to measure student and
teacher science content knowledge• Force and Motion• Flow of Matter and Energy in Living
Systems• Plate Tectonics
• Horizon Research• Assessing Teacher Learning About
Science Teaching (ATLAST)• www.horizon-research.com/atlast/• Instruments to measure student and
teacher science content knowledge• Force and Motion• Flow of Matter and Energy in Living
Systems• Plate Tectonics
ATLAST continuedATLAST continued
• Student tests - paper copies only• Teacher tests - paper or online versions• The student and teachers assessments
measure knowledge of the science content, while the teacher assessment also measures a teacher's ability to use science content knowledge to analyze student thinking, and to make instructional decisions. All of the items on the teacher assessment are set in instructional contexts.
• Student tests - paper copies only• Teacher tests - paper or online versions• The student and teachers assessments
measure knowledge of the science content, while the teacher assessment also measures a teacher's ability to use science content knowledge to analyze student thinking, and to make instructional decisions. All of the items on the teacher assessment are set in instructional contexts.
ATLAST continuedATLAST continued
• HRI provides the following analyses:• Percent of respondents choosing each
answer choice;• Percent of respondents answering each
item correctly;• Group mean score; and• Significance testing and computation of
effect sizes for pre- and post-test group mean scores.
• HRI provides the following analyses:• Percent of respondents choosing each
answer choice;• Percent of respondents answering each
item correctly;• Group mean score; and• Significance testing and computation of
effect sizes for pre- and post-test group mean scores.
MOSARTMOSART
• Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART)
• Harvard; PI - Phillip Sadler• Content instrument for K-12 physical
science and earth science, based on research about student misconceptions
• Completion of online tutorial (~30 min) required to download instruments
• Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART)
• Harvard; PI - Phillip Sadler• Content instrument for K-12 physical
science and earth science, based on research about student misconceptions
• Completion of online tutorial (~30 min) required to download instruments
www.cfa.harvard.edu/smgphp/mosart
MOSART continuedMOSART continued
• K-4 & 5-8 Physical Science, Earth Science, Astronomy/Space Science
• 9-12 Physics, Chemistry, Earth Science, Astronomy/Space Science
• Multiple versions of the test are available for pre/post administration
• K-4 & 5-8 Physical Science, Earth Science, Astronomy/Space Science
• 9-12 Physics, Chemistry, Earth Science, Astronomy/Space Science
• Multiple versions of the test are available for pre/post administration
MSPnet ToolboxMSPnet Toolbox
• http://hub.mspnet.org/index.cfm/msp_tools
• Clearinghouse for tools used and shared by MSP projects
• Tools may include assessment instruments, evaluation protocols, form letters, etc.
• http://hub.mspnet.org/index.cfm/msp_tools
• Clearinghouse for tools used and shared by MSP projects
• Tools may include assessment instruments, evaluation protocols, form letters, etc.
University of LouisvilleUniversity of Louisville
• Diagnostic Assessments for Middle School Teachers
• Each assessment is composed of 25 items—20 multiple-choice and 5 open-response.
• Six versions of each assessment are available in paper-and-pencil format
• Diagnostic Assessments for Middle School Teachers
• Each assessment is composed of 25 items—20 multiple-choice and 5 open-response.
• Six versions of each assessment are available in paper-and-pencil format
http://louisville.edu/education/research/centers/crmstd/diag_sci_assess_middle_teachers.html
University of Louisville (cont.)
University of Louisville (cont.)
• Assessments available free of charge• Optional scoring service for $10/teacher• Detailed summary of teachers'
performance that includes scores on individual items, on each science subdomain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)
• Assessments available free of charge• Optional scoring service for $10/teacher• Detailed summary of teachers'
performance that includes scores on individual items, on each science subdomain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)
University of Louisville (cont.)
University of Louisville (cont.)
Physical Science
Life Science Earth/Space Science
Matter Structure/Function Atmosphere/Hydrosphere
Motion & Forces Internal Regulation Lithosphere
Energy Heredity/Diversity Space
Interdependence
MSP Knowledge Management and
Dissemination
MSP Knowledge Management and
Dissemination• Knowledge reviews good for citing
research base• Searchable Instrument database
for science and math
• Knowledge reviews good for citing research base
• Searchable Instrument database for science and math
http://www.mspkmd.net/
Online Evaluation Resource Library (OERL)
http://oerl.sri.com/
Online Evaluation Resource Library (OERL)
http://oerl.sri.com/
• Teacher and student instruments• Content assessments, interviews,
surveys,classroom observations
• Teacher and student instruments• Content assessments, interviews,
surveys,classroom observations
Thinking About Mathematics Instruction
(TMI)
Thinking About Mathematics Instruction
(TMI)• http://www2.edc.org/tmi/
tmi_survey.html
• Survey to investigate elementary and middle school principals leadership content knowledge (LCK) for mathematics
• http://www2.edc.org/tmi/tmi_survey.html
• Survey to investigate elementary and middle school principals leadership content knowledge (LCK) for mathematics
Distributed LeadershipDistributed Leadership
• http://www.sesp.northwestern.edu/dls/instruments/
• Instructional Leadership Daily Practices Log
• Principal Experience Sampling Method (ESM) Log
• School Staff Network Survey
• http://www.sesp.northwestern.edu/dls/instruments/
• Instructional Leadership Daily Practices Log
• Principal Experience Sampling Method (ESM) Log
• School Staff Network Survey