Measuring Teacher Effectiveness
Laura Goe, Ph.D.Research Scientist, ETS
Principal Investigator for Research and Dissemination,
National Comprehensive Center for Teacher Quality
Oklahoma Department of EducationOklahoma City, OK August 15, 2012
2
Laura Goe, Ph.D.
• Former teacher in rural & urban schools Special education (7th & 8th grade, Tunica, MS) Language arts (7th grade, Memphis, TN)
• Graduate of UC Berkeley’s Policy, Organizations, Measurement & Evaluation doctoral program
• Principal Investigator for the National Comprehensive Center for Teacher Quality
• Research Scientist in the Performance Research Group at ETS
3
The goal of teacher evaluation
The ultimate goal of all teacher evaluation should be…
TO IMPROVE TEACHING AND
LEARNING
4
Evaluation for accountability and instructional improvement
• Effective evaluation relies on: Clearly defined and communicated standards
for performance Quality tools for measuring and differentiating
performance Quality training on standards and tools- Evaluators should agree on what constitutes
evidence of performance on standards- Evaluators should agree on what the evidence
means in terms of a score
5
Feedback (Coggshall et al., 2012)
“Feedback on whether or not instructional practices are working can come in the form of student learning data, the teachers’ own observations of student engagement, observations from a peer or a coach, a video-taped record of the practice, discussion within a professional learning community, or the results of a formal evaluation.” (p. 6)
Coggshall, J. G., Rasmussen, C., Colton, A., Milton, J., & Jacques, C. (2012). Generating teaching effectiveness: The role of job-embedded professional learning in teacher evaluation. Washington, DC: National Comprehensive Center for Teacher Quality.
http://www.tqsource.org/publications/GeneratingTeachingEffectiveness.pdf
6
Multiple Standards-based Measures of Teacher Effectiveness
• Affords many benefits to a comprehensive evaluation system Ability to triangulate results increases confidence in
evaluation outcomes More complete picture of teacher strengths and
weaknesses Each type of measure provides a different type of
evidence
• All work together to better inform professional development decisions
7
Using surveys for teacher evaluation
• Surveys are unlikely to show much change within a single year This limits usefulness in teacher evaluation as a
“pre-” and “post-” measure But they may be useful for revealing trends over
time For example, a new teacher is likely to improve
in many aspects of classroom management and discipline in the first few years, which is likely to be reflected in survey results
8
Surveys (2)
• The survey results will likely confirm what is already known through observations and other measures, but what is most important is what teachers do with the survey results Discussion with mentor, principal, colleagues Compare results with school, grade, or team
aggregates Support self-reflection & focus on areas in
need of improvement
9
Surveys (3)
• Almost no research on parent surveys• Research on peer surveys is all in higher
education• Research on student surveys is more
advanced, but only on one instrument—Harvard’s Tripod Survey Tripod was part of the Gates-funded
Measures of Effective Teaching study
10
Tripod Survey (1)
• Harvard’s Tripod Survey – the 7 C’s– Caring about students (nurturing productive relationships);– Controlling behavior (promoting cooperation and peer
support);– Clarifying ideas and lessons (making success seem
feasible);– Challenging students to work hard and think hard (pressing
for effort and rigor);– Captivating students (making learning interesting and
relevant);– Conferring (eliciting students’ feedback and respecting their
ideas);– Consolidating (connecting and integrating ideas to support
learning)
11
Tripod Survey (2)
• Improved student performance depends on strengthening three legs of teaching practice: content, pedagogy, and relationships
• There are multiple versions: k-2, 3-5, 6-12 • Measures:
student engagement school climate home learning conditions teaching effectiveness youth culture family demographics
• Takes 20-30 min• There are English and Spanish versions• Comes in paper form or in online version
12
Creating surveys (Over-simplified!)
• Convene a working group including teachers, principals, parents, students, etc.
• Determine which aspects of teaching standards can be measured with survey questions
• Create questions; try to ask some of the same questions in student, peer, parent, and self-assessment surveys for triangulation of responses
• Field-test the survey and have a psychometrician analyze results (IRT model)
• Make adjustments and pilot the survey (no stakes); then make final revisions
13
Using service learning for teacher evaluation
• Service learning has the potential to be useful in teacher evaluation, particularly when trying to measure teachers’ contributions to student learning in non-tested subjects and grades (lower elementary, high school, subjects that are not part of statewide testing)
• The goal should be comparability in results across students and teachers within a district
As much as possible, teachers within the same subject/grade should do the same service learning projects and use the same assessments
14
Service learning (2)
• Great value in service learning Teachers collaborating around student learning across
curriculum and grades Most subject/grade standards can be measures with a
service learning project Learning goals and assessments can be differentiated
for special needs students and ELLs
• This makes it important to select a service learning project that is wide in scope rather than narrow
• May incorporate “place-based” learning strategies Students and teachers strengthen relationships with local
community and learn about local geography and history
15
Service learning (3)
• There is considerable research on service learning, but mostly on outcomes related to the projects
Not a lot of research on student growth that could be directly attributed to teaching practices
• A good service learning project would have multiple measures of student learning over time (pre- and post-) and may include multiple subjects
• Could take the form of a student portfolio scored with a rubric
Scoring processes, audits, etc. would need to be put into place for validity and comparability
16
Creating a service learning template and guidance (oversimplified!)
• Convene a working group including teachers, principals, parents, students, & community leaders
• Determine several broad-scope projects that could take place across a semester or a school year
Using these projects as examples, identify the student learning standards that could be assessed
Identify measures that could show student learning growth on standards (portfolio entries, tests, performances, presentations, group projects, written reports/journals, etc.)
• Field-test projects and measures and analyze results—use to create guidance docs & templates
17
Service learning for teacher evaluation purposes: A possible approach
• Teacher teams and principals meet with district staff to select a service learning project that all or most teachers will participate in throughout the district
• Using state guidance doc and templates, teachers and principals develop plan and timeline for projects
Teachers from grades and subjects work together in teams to select appropriate assessments to measure student learning
• Using state guidance, assessments are scored and compiled and teachers’ contributions to student learning growth can be determined
18
Resources on Service Learning
Bradley, L. R. (2006). Service-Learning: A strategy for school improvement and building effective community partnerships. Columbus, OH: Ohio Department of Education.
http://www.servicelearning.org/filemanager/download/Ohio_Department_of_Education_training_manual.pdf
Colorado Department of Education. (2004). Creating high-performing schools through service-learning: A service-learning trail guide. Denver, CO: Colorado Department of Education.
http://www.ecs.org/clearinghouse/57/13/5713.pdf
Davis, K. M., Miller, M. D., & Corbett, W. T. (1998). Methods of evaluating student performance through service learning. Gainesville, FL: College of Education, University of Florida.
http://www.fsu.edu/~flserve/resources/methods/MethodsEvaluatingStudentPerformanceThroughSL.pdf
Ernst, J. A. (2004). Prairie Science Class evaluation report. Fergus Falls, MN: Prairie Wetlands Learning Center.
http://meera.snre.umich.edu/sites/all/files/Prairie_Science_Class_Evaluation_Report.pdf
19
Teacher portfolios
• It’s not a “brag book,” it’s evidence of how the teacher sets/pursues meaningful goals related to improving knowledge & instructional practice
• More like a year-long self-exploration of one or more standards related to aspects of teacher’s practice such as:
Restructuring the classroom learning environment to support diverse learners’ needs
Working with a collaborative team to improve reading outcomes
Developing expertise in classroom technology to support new methods of instruction
20
Portfolios (2)
• Portfolios do not need to be “physical” binders but can be virtual evidence collections
This is easier for most teachers to create and share and easiest for others to review
Allows teachers to include multiple types of data (videos, spreadsheets, recordings, photographs, work samples, planning documents, self-reflection, peer comments, etc.)
• Teachers should receive training in Choosing standards to focus on Choosing a way to demonstrate growth on the standards Selecting evidence to include in the portfolio
21
Portfolios (3)
• State’s role will be to provide guidance docs, templates, and examples of virtual portfolios at various score levels (like “unsatisfactory” and “exemplary”)
• Principals and others involved in scoring portfolios will need to be trained to score portfolios with accuracy and consistency in order for results to be valid indicators for teacher evaluation purposes
• Teachers and principals should be involved in the development, and the process should be piloted (no stakes)
22
Some “popular” observation instruments
Charlotte Danielson’s Framework for Teaching
http://www.danielsongroup.org/theframeteach.htm
CLASS
http://www.teachstone.org/
Kim Marshall Rubric
http://www.marshallmemo.com/articles/Kim%20Marshall%20Teacher%20Eval%20Rubrics%20Jan%
Marzano Teacher Evaluation Framework
http://www.marzanoevaluation.com/
23
Example: University of Virginia’s CLASS observation tool
Emotional Support Classroom Organization
Instructional Support
Pre-KandK-3
Positive Climate
Negative Climate
Teacher Sensitivity
Regard for Student (Adolescent) Perspectives
Behavior Management
Productivity
Instructional Learning Formats
Concept DevelopmentQuality of FeedbackLanguage Modeling
Upper Elementary/Secondary
Content UnderstandingAnalysis and Problem
SolvingQuality of Feedback
24
Domain 1: Planning and Preparation includes comprehensive understanding of the content to be taught, knowledge of the students’ backgrounds, and designing instruction and assessment.
Domain 3: Instruction is concerned with the teacher’s skill in engaging students in learning the content, and includes the wide range of instructional strategies that enable students to learn.
Domain 2: The Classroom Environment addresses the teacher’s skill in establishing an environment conducive to learning, including both the physical and interpersonal aspects of the environment.
Domain 4: Professional Responsibilities addresses a teacher’s additional professional responsibilities, including self-assessment and reflection, communication with parents, participating in ongoing professional development, and contributing to the school and district environment.
Example: Charlotte Danielson’s Framework for Teaching
25
Domain 1 from “The Four Domains of the Marzano Teacher Evaluation Model”
26
Example: Kim Marshall’s Rubric
Planning & Preparation for Learning
Highly Effective Effective Improvement Necessary
Does Not Meet Standards
a. Knowledge
Is expert in the subject areaand has a cutting-edge graspof child development and howstudents learn.
Knows the subject matter welland has a good grasp of childdevelopment and howstudents learn.
Is somewhat familiar with thesubject and has a few ideas ofways students develop andlearn.
Has little familiarity with thesubject matter and few ideason how to teach it and howstudents learn.
b. Strategy Has a well-honed game planfor the year that is tightlyaligned with state standardsand assessments.
Plans the year so students willmeet state standards and beready for externalassessments.
Has done some thinking abouthow to cover high standardsand test requirements thisyear.
Plans lesson by lesson and haslittle familiarity with statestandards and tests.
27
Creating an observation instrument
• Consider the research: lots of evidence on use of Framework for Teaching and CLASS, little for other instruments
Don’t reinvent the wheel
• Working group to design instrument should include teachers, principals, other stakeholders
• Focus on teaching standards What evidence will you need to be able to score the
teacher’s performance on this standard?
• Field test and pilot instrument• Training for scoring is crucial for valid results
28
Questions?
29
Laura Goe, [email protected]://twitter.com/GoeLaura
National Comprehensive Center for Teacher Quality1000 Thomas Jefferson Street, NWWashington, D.C. 20007www.tqsource.org
Top Related