1

Washington State Teacher and Principal

Evaluation Project

Conducting High-Quality Observations

June 2013

2

As you enter, please have a brief discussion with your district team to decide your level of confidence in whether the following statements are true for your district:1. Our evaluators know how to minimize bias when

collecting observation data.2. Our evaluators know how to capture observation

evidence using a running record approach to note-taking.

3. Our teachers trust our observers to capture accurate evidence that will provide them with useful feedback for professional growth.

Write your district name on three sticky notes and place them on the confidence scales posted on [INSERT LOCATION] for each statement.

Entry Task: Confidence Conversation

3

Introductions Logistics Agenda

Agenda Connecting Learning I Learning II Learning III Learning IV Reflecting Wrap-Up

Welcome!

4

Introduction to Educator Evaluation in Washington Using Instructional and Leadership Frameworks in

Educator Evaluation Preparing and Applying Formative Multiple Measures

of Performance: An Introduction to Self-Assessment, Goal Setting, and Criterion Scoring

Including Student Growth in Educator Evaluation Conducting High-Quality Observations and

Maximizing Rater Agreement Providing High-Quality Feedback for Continuous

Professional Growth and Development Combining Multiple Measures Into a Summative

Rating

Modules

5

The Evaluation System Components

6

Pausing Paraphrasing Posing Questions Putting Ideas on the Table Providing Data Paying Attention to Self and Others Presuming Positive Intentions

What Else?

Session Norms

7

Connecting

Builds community, prepares the team for learning, and links to prior knowledge, other modules, and

current work

8

Conducting High-Quality Observations Maximizing Rater Agreement

Reminder! This module provides an orientation to

the basic concepts. This module does not go into great depth

about evidence relating to any of the specific instructional or leadership frameworks and instead leaves it up to the districts to seek additional training.

Module Overview: 2 Parts

9

Participants will know and be able to: Describe the role of observation as a

methodology for gathering evidence of educator performance and its role in achieving rater agreement.

Utilize the revised WACs to identify the number and length of observations for different categories of staff.

Identify common observation biases in their own and others’ practice.

Utilize appropriate strategies for minimizing bias in the observation.

Utilize best practices in collecting observation data.

Overview of Intended Participant Outcomes

10

Remember: Observation is one measure within a comprehensive evaluation system that includes multiple measures, including the following: Self-Assessment Goal-Setting Artifact Review

Observation as One Piece of the Puzzle

Observation is another piece, helping us move closer to enacting TPEP’s Core Principles and the new system

11

“We Can’t Fire Our Way to Finland” The critical importance of teacher and leadership

quality The professional nature of teaching and leading a

school The complex relationship between the system for

teacher and principal evaluation and district systems and negotiations

The belief in professional learning as an underpinning of the new evaluation system

The understanding that the career continuum must be addressed in the new evaluation system

The system must determine the balance of “inputs or acts” and “outputs or results”

TPEP Core Principles

12

Rater Agreement Depends on High- Quality Evidence Collection Practices

Standards

Criterion 2

Criterion 1

Criterion 3

Criterion 4

Criterion 5

Criterion 6

Criterion 7

Criterion 8

Frameworks+

Student Growth Rubrics

Evidence• Observatio

n• Student

growth data

• Artifacts• Other

evidence relevant to the frameworks

Criterion Rating

District-determined process

DistinguishedProficientBasicUnsatisfactory

Summative RatingState-determined process

DistinguishedProficientBasicUnsatisfactory

Student Growth Measures

(From 3 specific criteria)

Student Growth Impact Ratings:

Low, Average, High

13

The extent to which the scores between the raters exhibit consistency and accuracy against predetermined standards. The predetermined standards are the instructional and leadership frameworks and rubrics that define the basis for summative criterion-level scores.

Rater Agreement

14

Evaluators need multiple opportunities and settings to observe and assess educator practice.

Multiple observations paired with timely feedback are a key part of a strong evaluation system.

Using multiple observers and multiple observations improves the reliability of scores more than having longer observation periods.

Using a mix of shorter and longer observations can decrease costs while maintaining increased reliability of scoring.

Using a mix of internal and external observers (school-based and not) can help determine the validity of practice scores.

(Sources: Ho & Kane, 2013; Sartain et al., 2011)

What the Research Says…

15

Handout 1: Studies link classroom observations to student achievement, by Stephen Sawchuck

Handout 2: Feedback for Better Teaching, by The Measures of Effective Teaching (MET) Project. (pp. 1-2 and 6-7; skip pp. 3-5)

Voices on Observation

16

What are the key messages and ideas in the two articles with regard to observations?

Based on what we just read, what are some best practices for observation?

Voices on Observation Wrap-Up

17

Learning

Understand the new requirements for educator observation in Washington, as well as best practices

and strategies for conducting high-quality observations

18

1. In the 2012-2013 school year, Mr. Davis was in his eighth year of teaching at your school. He has continuing contract status and was rated “satisfactory” in his last four evaluations.

True or False: Mr. Davis is eligible for focused evaluation in 2013-2014.

Answer: True RCW 28A.405.100 (12)(b)(iv)(c)(i)

Learning Introduction: Meet the New Rules

19

2. Fill in the blank: Mr. Garvey has been a principal in North Thurston for the past five years. He is starting his first year as a principal in your district in September. Mr. Garvey should have a ___________ evaluation [focused or comprehensive] this year.

Answer: ComprehensiveRCW 28A.405.100 (12)(b)

Learning Introduction: Meet the New Rules

20

3. Fill in the blank: Mr. Burke has taught for ten years in your district and has continuing contract status. He is currently on a comprehensive evaluation cycle. Mr. Burke must be observed ____ times this year for no less than _____ minutes total.

Answer: 2 and 60WAC 392-191A-070

Learning Introduction: Meet the New Rules

21

4. Fill in the blank: Mrs. Wright is a 20-year veteran teacher but just started teaching at your school this year. As a new employee, Mrs. Wright must be observed once for ___minutes during the first ____calendar days of the employee period.

Answer: 30 and 90WAC 392-191A-070 (2)

Learning Introduction: Meet the New Rules

22

5. Mr. Jones is starting his third year in teaching and is a provisional teacher. This year, he must be observed: A. At least two times for a total of 120 minutes B. At least three times for a total of 90 minutes C. At least four times for a total of 150 minutes

Answer: BWAC 392-191A-070 (3)

Learning Introduction: Meet the New Rules

23

6. Ms. Thomas has been on a focused evaluation for the past two years. In both years, she was rated as proficient. This year: A. She must have a comprehensive evaluation. B. She and her supervisor can decide whether to

continue with a focused evaluation or switch to a comprehensive evaluation.

C. She must have a focused evaluation.

Answer: BWAC 392-191A-070(12)(c)(iv)

Learning Introduction: Meet the New Rules

24

7. Mr. Tucker had a comprehensive evaluation last year and was rated as “proficient”; however, he and his evaluator identified Criteria 4 as an area that needed improvement.

True or False: Mr. Tucker is required to use Criterion 4 for his focused evaluation next year.

Answer: FalseWAC 392-191A-120(2)

Learning Introduction: Meet the New Rules

25

8. Mr. Tanyu has continuing contract status and is on a focused evaluation cycle. He selected Criteria 7: “Communicating and collaborating with parents and school community.” You notice that Criteria 7 does not need to be assessed through observation; other methods, such as artifact review, may be more appropriate.

Mr. Tanyu: A. Does not need to be observed this year. B. Must be observed this year. C. Districts can decide whether or not to require observation in

this situation.

Answer: FalseWAC 392-191A-110

Learning Introduction: Meet the New Rules

26

9. Part I: Principal Smith completed an observation of Ms. Williams on Tuesday and promptly wrote up the results of the observation later that day. Principal Smith must give a copy of the report to Ms. Williams within: A. four days. B. no more than one week. C. three days.

9. Part II: Principal Smith and Ms. Williams must hold a minimum of ____ confidential conferences during each year.

Answers: C and 2WAC 392-191A-130

Learning Introduction: Meet the New Rules

27

10. True or False? The purpose of observation is to: (a) collect accurate, relevant evidence to evaluate

against instructional and/or leadership frameworks, and

(b) provide educators with quality feedback.

Answer: True!

Learning Introduction: Meet the New Rules

28

Frequent: research agrees that a single observation of teacher or leader practice is insufficient for capturing evidence to generate accurate and reliable ratings.

Comprehensive: captures evidence appropriate for assessing the educator against the required criteria.

Accurate: minimizes bias and error to ensure that collection and review of the data produces an authentic assessment of the educator’s performance.

Useful & Timely Feedback: generates detailed, focused feedback for improvement.

Learning Content I: High-Quality Observation

29

What does it mean to be [INSERT DESCRIPTOR] in your observations?

How does this principle of high-quality observations better support the improvement of teaching and learning?

Learning Content I Activity: High-Quality Observation

30

Share out your two key ideas for your Guiding Principle

Frequent

Comprehensive

Accurate

Feedback

Learning Content I Activity: Debrief

31

A skilled observer: 1. Understands each component and indicator on

the district rubric thoroughly and deeply.

2. Gathers and sorts sufficient evidence of practice as it happens in the classroom or school.

3. Recognizes and puts aside preferences and biases.

4. Interprets the evidence appropriately to give an accurate rating using the evaluation instrument.

Learning Content II: Strategies for Reducing Bias in Observation

(McClellan, Atkinson, & Danielson, 2012)

32

Bias enters the process in two places:

Observation bias: while recording data during an observation, every observer has a “lens” altered by: Specializations: content or grade level Personal instructional preferences Past observations or impressions of the educator Individual bias or experiences.

Rater error and bias: observers rate performance too high or too low based on considerations other than the instructional or leadership framework.

Common Sources of Bias in Observation

33

Record evidence, not judgment or opinion: Evidence describes a concrete behavior, practice,

interaction, or material that can be used to support a rating decision and provide clear feedback.

Judgment or opinion comments on or interprets what is being observed but does not provide concrete evidence.

Observation notes are written as evidence statements:

Evidence statement: A factual statement, free from opinion or bias, that describes teacher practice.

Reducing Bias in Observation

34

EXAMPLE:Opinion Statement: “Yes! Good questioning, engages the students.”

Evidence Statement: T: “How do you know, or how did you get that answer?” S: (Multiple hands shoot up) T: Calls on AlissaS: “I went back and used Pythagorean’s formula.” S writes out process on board and explains each step.

Reducing Bias in Observation

35

Record: Direct quotes Questions asked Observed actions or movements Literal descriptors

Avoid: Jotting down extraneous information Jotting down initial thoughts on ratings

Strategies for Recording Evidence

36

Examine Handout 4

Highlight factual statements

Underline judgment statements or statements not based on evidence

Learning Activity II: Evidence Versus Opinion

37

Holding the Pre-observation Conference Review the proposed lesson plan Listen to the teacher’s perspective and learn

about the classroom context Understand the teacher’s professional goals and

expectations Identify any specific areas the teacher would like

feedback on

Learning Content III: Best Practices for Collecting Observation Data

38

Before Observing: Focus: Review pre-conference notes and the

educator’s professional goals

Clean Your Slate: Purposefully identify and set aside previous impressions from earlier observations

Best Practices for Collecting Observation Data

39

While Observing: Be “a fly on the wall”: Avoid interacting with the

participants to minimize your influence on what you are observing.

Record only information you actually observe—do not “fill in the blanks” or assume other practices are occurring without actually observing them.

Record the depth and frequency of key “look-fors” (e.g., types of questioning, student behaviors, re-direction to task).

Best Practices for Collecting Observation Data

40

Balance taking detailed notes with observing the action in front of you. Use abbreviations to increase efficiency (e.g.,

“T” for teacher, “S” for student, or components by their first two letters—“LD” for lesson delivery).

Paraphrase around short quotes to convey the spirit and tone of the interaction.

Best Practices for Collecting Observation Data

41

41

Example Observation Notes

Best Practices for Collecting Observation Data

42

Learning Activity III: Practicing Collecting Observation Data

Video Background “Ms. K” is teaching a sixth-grade science

lesson. 25 students, ranging from 11 to 13 years old,

are in her class. Two students are English language learners

and several students have IEPs for accommodations in reading.

NBPTS Video #161: Probing Student Understanding(http://198.101.195.20/video.php?p=15346487)

43

Compare your notes for the following: Step 1: Do the notes capture the same data?

What are you missing? Step 2: Do the notes record evidence, such as:

quotes? questions? actions or movements?

Step 3: Do the notes include judgments or opinions? Step 4: Did you identify any personal biases while you

were taking notes?

Learning Activity III: Practicing Collecting Observation Data

44

How many of you know what eVAL is? Is anyone currently using eVAL? Is anyone considering using eVAL?

Learning Content IV: Overview of eVAL

45

A web-based tool designed to manage the evaluation process and documentation

A free resource! Personalized for each district for their

instructional framework, resources, and documents

Voluntary for all districts, who can use as many or as few of eVAL’s features as they’d like (or none at all)

Extremely secure, with limited access physically and virtually to its servers

eVAL

46

eVAL helps evaluators and educators in five specific ways: Practice: Enter formative evaluation data

regarding videos or classroom observations and engage in a dialogue about shared (or different) perceptions of quality instruction.

Conference: Plan for pre-conferences, observations, and post-conferences—including sending questions in advance of meetings and scheduling observations.

eVAL

47

Observe: Principals can… Enter notes during observations (or upload them

from other software). Align evidence from notes to instructional and

leadership frameworks and provide feedback to teachers.

Score and annotate the rubric and manage all aspects of evidence gathered in the observation.

Create post-conference observation reports.

eVAL

48

Summary & Reports: Principals can… Submit final summative scores for teachers to their

district office. Reflect on all the evidence gathered for the year. Score final summative rubrics. Print final summative reports, including schoolwide

evaluation summaries, individual evaluation trends, and discrepancies between teacher self-assessments and evaluator scores.

eVAL

49

For more information, check out the eVAL website: http://tpep-wa.org/resources/eval/

Watch eVAL video walkthroughs: http://tpep-wa.org/resources/eval/eval-video-walkt

hroughs/

eVAL

50

Reflecting

51

Select one of the Debrief Cards and share:

One key observation I made during this session

One key question I have as a result of this session

One way someone else helped me out during this session

Something new I have learned as a result of this session

One feeling I experienced during this session

Whip Around and Plus/Delta Debrief

52

Wrap-Up

Understand the options for homework

Preparation for the Maximizing Rater Agreement module

53

Bring a copy of: Your observation notes from Learning Activity III. Your district’s instructional framework.

You will use these to become familiar with best practices in analyzing observation data and achieving rater agreement.

Preparation for the Maximizing Rater Agreement Module

54

Discuss and plan an annual schedule of observations: Consider the number of evaluators, the number of

teachers, and the number of school days. Consider how many teachers and principals are on

focused versus comprehensive evaluations. Consider setting deadlines for completing the

observation of all teachers.

Suggested Homework

55

Thank you!

Presenter Name

xxxxxxxxxxx@xxx.xxx