Module 4 for Administrators

The Real Implications of VAM and Florida Standards

for Teacher Evaluation

2

Professional Development Session Alignment

Set 1Governing Board

School Leaders

Module 6 FL CCRS Math

Teachers Math

Leadership Teams Session 2

Session1

ELAData Use

Data Use ELA Math

Data Use

3

Professional Development Session Alignment

Set 2Governing Board

School Leaders

Module 7 ELA & Data Use

Module 8 Math & Data Use

Teachers Math

Leadership Teams

Session 4

Session3

ELA

AssessmentVAM &

Academic Rigor

Florida Standards

Data &ELA

Data &Math

Session 5

Session 6

4

Data

5

8 Components of Full Florida Standards Implementation

Develop an understanding of Florida’s Value-Added Model (VAM) and its implications to teachers as well as the entire charter school.

Increase awareness of the higher standards of learning and rigor the new assessments will require of students and staff.

Recognize best practices that teachers should be implementing to increase rigor in the classroom.

Develop an understanding of teacher and student behaviors that should be observed during classroom walkthroughs to ensure a rigorous classroom.

Recognize the importance of Webb’s Depth of Knowledge Levels and its connection to increased rigor.

Module Outcomes

6

Today’s Agenda

• Welcome and Introductions• Pre-Assessment• Section 1: Teacher Evaluation- The Value-

Added Model• Lunch• Section 2: Understanding Academic Rigor• Section 3: Connecting Academic Rigor

and Depth of Knowledge• Next Steps• Wrap Up and Post-Assessment

7

To Do List

Pre-Assessment

Introductory Activity

8

Guide Page

4

Teacher Evaluation: The Value-Added Model (VAM)

9

Section 1

• Research has shown that the best predictor of a teacher’s effectiveness is his or her past success in the classroom. Most other factors pale in comparison, including a teacher’s preparation route, advanced degrees, and even experience level (after the first few years).*

• The lesson is clear: to ensure that every child learns from the most effective teachers possible, schools must be able to gauge their teachers’ performance fairly and accurately.

*Jordan, Mendro, and Weerasinghe, Teacher Effects on Longitudinal Student Achievement, 1997

Why Change Teacher Evaluation Systems?

10

• “The effect of increases in teacher quality swamps the impact of any other educational investment, such as reductions in class size.” Goldhaber, 2009

• “More can be done to improve education by improving the effectiveness of teachers than by any other single factor.” Wright, Horn and Sanders, 1997

• “Having a high-quality teacher throughout elementary school can substantially offset or even eliminate the disadvantage of low socio-economic background.” Rivkin, Hanushek and Kain, 2005

Teachers Matter

11

Changing the Traditional Teacher Evaluation Model to Ensure Quality

12

Using Multiple Measures

Watch the video on changes to teacher evaluation systems. Discuss with members at your table the challenges and opportunities for your charter school with stronger teacher evaluation models.

What Changes to the Florida Teacher Evaluation System Have Been Established?

• Established by Student Success Act (SB 736) Educator Evaluation System

• Teacher and School Administrator Evaluations

Professional Practice Student Learning Growth

• FLDOE Student Growth Implementation Committee (SGIC)

13

What Is the Biggest Shift In Florida’s Teacher Evaluation System?

14

The Teacher Evaluation Model Now Has Two Major Components: Instructional Practice

• Deliberate Practices• Teacher Observation• Professional Growth• Surveys

Student Performance (VAM data)

Important to Know….It is important to note that measures used in the Florida School Grading system, student proficiency and learning gains, are not a part of the Florida Value-Added Model of student learning growth.

Student learning growth is based on the actual scale score on FCAT 2.0 tests. Although students may score in Achievement Levels 3-5, they still have room to grow.

15

• A Value-Added model attempts to measure the impact of a teacher on student learning, by accounting for other factors that may impact the learning process.

• The models do not:• Evaluate teachers based on a

single year of student performance or proficiency.

• Evaluate teachers based on simple comparisons from one year to the next.

The New Measure: Value-Added Analysis

16

Let’s Take a Break…

17

Be back in 15 minutes…

Comparing the Features of a Growth Model to a Value-Added Model

Understanding the Purpose of Student Growth Models

18

Looking at the Difference Between Growth and Proficiency

Growth (Progress)Growth models measure the amount of academic progress students make between two points in time

Proficiency (Status)A method for measuring how students perform at one point in time

19

Looking at the Difference Between Growth and Proficiency

Stu A Stu B Stu C Stu D Stu E0

50100150200250300350400450500

Prior Current

Stu F Stu G Stu H Stu I Stu J0

50100150200250300350400450500

Prior Current

Growth?

Teacher 1 Teacher 2

ProficiencyProficiency

Growth

20

Growth> Compare student to

own prior performance > Consider student

characteristics> Progress between

points> Critical to student

success

Proficiency> Compare student to

a standard> Does not consider

student characteristics> Performance at a

point in time> Critical to

postsecondary opportunity

A Better Picture of Student Learning

21

&

VAM & Student Learning Growth

• Teacher contribution to a change in a student’s achievement on standardized test

• Calculated from a measure of student learning growth over time

• Students may not be proficient, but may have met their student growth target

http://www.fldoe.org/committees/sg.asp22

23

Understanding VAM: What Are The Scores?What is the Predicted Student Score? • It is the score you would EXPECT a student to achieve based on the

student’s performance on prior tests and other information available on the student.

• A predicted score for a student is generated based on what would normally happen in an average class with a typical teacher.

What is the Student Learning Growth Score? • The difference between current test score and predicted test

score.

Student Learning Growth is the Amount Above or Below Predicted Score

24

Proficiency vs. Predicted Growth (VAM)

Stu A Stu B Stu C Stu D Stu E0

50100150200250300350400450500

Prior Current PredictedStu F Stu G Stu H Stu I Stu J

0

50

100

150

200

250

300

350

400

450

500

Prior Current Predicted

Teacher 1 Teacher 2Did Student B meet or exceed the predicted score?

Did Student J meet or exceed the predicted score?

25

• Teachers teach classes of students who enter with different levels of proficiency and possibly different student characteristics.

• Value-Added models ATTEMPT to “level the playing field” by accounting for differences in the proficiency and characteristics of students assigned to teachers.

26

Advantages of Value-Added Models

• Value-Added models are designed to MITIGATE the influence of differences among the entering classes so that schools and teachers do not have advantages or disadvantages simply as a result of the students who attend a school or are assigned to a class.

• Value-Added models are not perfect. The model will be continually reviewed by the FLDOE in case adjustments are necessary.

Focusing on Student Growth1. Discuss with others at your table the impact focusing on

student growth rather than proficiency will have at your school.2. In the Participant Guide on page 6, review the guiding

questions. Brainstorm ways that using a student growth reporting system will be more informational and beneficial than the traditional proficiency based system that is currently in place.

3. Be prepared to share with the entire group.

Activity 1: Focusing on Student Growth

27

Guide Page

6

STUDENT Characteristics

CLASSROOM Characteristics

SCHOOL Characteristics

Understanding VAM: Factors Used To Adjust Predicted Score

28

Factors Identified to “Level the Playing Field”

Understanding VAM: Student Characteristics

29

Students with Disabilities

English Language Learners Gifted Status

Attendance MobilityNumber of

Subject-Relevant Courses Enrolled

Two Years Achievement

Scores

Difference from Modal Age in

Grade

30

Understanding VAM: Factors NOT Used To Adjust Predicted Score

30

Gender Race

EthnicitySocio-

economic Status

Student Characteristics NOT directly accounted for

Since these factors already influence a student’s performance and prior performance is the predictor with the strongest weight, they are indirectly accounted for.

Student A

Prior FCAT Reading

Score: 1700

Attendance: 10 days absent

English Language Learner

Student B

Prior FCAT Reading

Score: 1700

Attendance: 0 days absent

Not an English Language Learner

How Do The Factors Affect The Predicted Scores? – An Example

31

Student A Predicted

Score1750

Student B Predicted

Score1790

How Are Student Growth ScoresCalculated for the Teacher/School?

• State calculates scores for each grade and subject separately

• Therefore, a teacher/school may have more than one score

32

Percent Ranks and Ratings• State generates a percent rank for each

teacher, school, district based currently on FCAT scores• Reading score is ranked among all

Reading scores in the state (Grades 4-10)

• Math score is ranked among all Math scores in the state (Grades 4-8, No Algebra EOC)

• Reading + Math score is ranked among Reading + Math scores in the state

33

What Are the Key Features of A Teacher Report for VAM Data?

34

35

What Are the Key Features of A Teacher Report?

36

What Are the Key Elements of A Class Report?

Amount of growth above predicted score made by student

10 out of 13 students met their expectation

Sample School Report for Students

37

Effective

Combined Scores

Sample Questions to Ask When Analyzing VAM Data

What are our strengths as a charter school?

• In what grade levels and content area(s) did we produce more than expected growth with our students?

• Is there anything special or different about the instructional strategies when delivering that content?

What are the challenge areas for our charter school?

• What is different about how this content is delivered?

• What root cause can we determine that may affect the progress in that content area?

38

Activity 2: Understanding VAM and Its Impact on Charter Schools

39

Understanding VAM: the Value-Added Model 1. Look at the table on page 7 in your Participant Guide. The table

has columns labeled: New Information, Opportunities, Challenges, and Questions for your District regarding the new teacher evaluation system which includes VAM.

2. Individually, using all the information (including new learning) fill out the table as it relates to your charter school.

3. After you have completed the table, share with other participants your thoughts and discuss the reflection questions at the bottom of the activity sheet.

Guide Page

7

Bon Appétit

40

Be back in 1 hour…

Understanding Academic Rigor

41

Section 2

How Can School Leaders Support Stronger VAM Scores?

42

• School leaders cannot change how VAM scores are calculated, but do have influence over the level of rigor in classrooms in their charter school.

• Focusing on what teachers and students are doing in the classroom will raise student achievement level and eventually affect VAM scores.

Are our students ready for more rigor?

What is Rigor?

Quickly jot down your definition of rigor and characteristics of a rigorous lesson you may see at your school. In your table group, discuss your definition with others and determine if there is a consensus.

Think of a teacher who applies rigorous practices in their classroom.

Defining Academic Rigor

43Teaching What Matters Most: Standards and Strategies for Raising Student Achievement. ASCD, 2001

“When instruction is academically rigorous, students actively explore, research and solve complex problems to develop a deep understanding of core academic concepts that reflect college readiness standards.” Oregon Small Schools Initiative

“Rigor is creating an environment in which each student is expected to learn at high levels; each student is supported so s/he can learn at high levels; and each student demonstrates learning at high levels.” Barbara Blackburn

What Is Academic Rigor?

44

Teaching What Matters Most: Standards and Strategies for Raising Student Achievement. ASCD, 2001

Rigor 101 With Barbara Blackburn

Low Expectations

Less Challenging Content-Courses

Low Level Instruction

Assignments

Poor Assessment

Results

Without Academic Rigor: Cycle of Low Expectations

45

Assessing Evidence of Academic Rigor In Your Charter School1. Review the list of standards of academic rigor in schools on pages 9-11

in your Participant Guide.

2. For each standard, assess the level of rigor (1-3) you believe best reflects your school at this time. Be sure to include evidence that supports your implementation level.

3. Once you have completed the survey, discuss the reflection questions on the Activity Sheet with members of your table group.

Activity 3: Assessing Evidence of Academic Rigor In Your Charter School

46

Guide Pages 9-11

What Does Academic Rigor Look Like In the Classroom?

Rigor Is Not…

• Something extra you assign on top of everything else

• More of the same math problems• Faster rate of instruction• More homework

Rigor Is…

• An integral part of quality instruction

• Increasing the complexity of questions that require more than one word answers

• Expanding academic challenges• Digging deeper into an

assignment to gain full understanding

47

What Should

Teachers Do?

Create an environment to support student learning at high

levels.

Ask higher order thinking

questions.

Support students with gaps, not

provide different assignments.

Assess students to allow them to

demonstrate their knowledge.

What Should Teachers Be Doing In A Rigorous Classroom?

48

Davis, P. (2013). Teaching to the Rigor of the CCSS. ASCD Webinar

What Should

Students Do?

Think deeply about tasks.

Interpret information &

synthesize knowledge.

Analyze their learning and

apply it.

Challenge themselves

intellectually.

What Should Students Be Doing In A Rigorous Classroom?

49

Davis, P. (2013). Teaching to the Rigor of the CCSS. ASCD Webinar

Getting Rigor Right, What You Should See In A Classroom1. Read the article “

Before and After the Walkthrough: What to Do to Improve Instructional Rigor”, which is a separate handout.

2. After reading the article, review the eight best practices highlighted in the article.

3. Use the Activity Sheet on pages 12-13 of your Participant Guide to assess the practices currently used at your charter school to raise the rigor of instruction to better align to the Florida Standards.

4. Determine which areas your school has been making strides to accomplish and which areas need more focus and direction. Think about if the best practices for rigor are reflected in your Florida Standards Implementation Plan. Discuss with others at your table.

Activity 4a: Getting Rigor Right, What You Should See In a Classroom

50

Guide Pages 12-13

Activity 4b: Assessing Rigor During A Lesson

51

Assessing Rigor During A Lesson

1. Watch the video of a high school class teaching a lesson on the Declaration of Independence.

2. Using the Focusing on Rigor Classroom Walkthrough Tool on pages 14-16 in the Participant Guide, assess the level of rigor during the lesson. Be sure to include evidence.

3. After the video, discuss with your table groups the observations made regarding the rigor of the lesson.

4. As a group, outline feedback that you as a school leader would provide to the teachers in the class to increase the rigor of the lesson.

Teaching the Declaration of Independence

Guide Pages 14-16

Connecting Academic Rigor and Depth of Knowledge

52

Section 3

Rigor’s Impact on Assessment

53

• Rigor in assessment is important for students to demonstrate what they know and are able to do.

• Assessing students’ depth of knowledge in a content area aligns to the expectation of the new Florida Standards and assessments.

The new Florida Standards increases Rigor and Depth of Knowledge.

Student’s Depth of Knowledge should focus on the complexity of the Florida Standards in order to successfully complete an assessment or task. The outcome is the focus of the depth of understanding.

Assessments must also accurately measure students’ depth of knowledge of the content standards.

Aligning Rigor and Depth of Knowledge (DOK) with Curriculum, Instruction, and Assessment

54

• Focuses on the complexity of content standards in order to successfully complete an assessment or task. The outcome (product) is the focus of the depth of understanding.

• Aligns to the rigor of the new Florida Standards and the higher expectations that teachers need to have for student learning.

Why Depth of Knowledge?

55

• Norman Webb (1997) developed a process and criteria for systematically analyzing the alignment between standards and assessments.

• Depth of Knowledge (DOK) is in reference to the complexity of mental processing that must occur to answer a question, perform a task, or generate a product.

• DOK measures cognitive demand, or complexity, and is assigned using Levels 1 to 4.

Looking At Webb’s Depth of Knowledge

56

57

Webb’s Four Levels of Cognitive Complexity

• Recall and ReproductionLevel 1• Skills and ConceptsLevel 2• Strategic ThinkingLevel 3• Extended ThinkingLevel 4

Depth of Knowledge Overview

58

• Requires recall of information, such as a fact, definition, term, or performance of a simple process or procedure

• Answering a Level 1 item can involve following a simple, well-known procedure or formula

DOK Level 1: Recall and Reproduction

Discuss at your table what DOK 1 would look like in a classroom. What would the teacher be doing, what would students be doing?

• Includes the engagement of some mental processing beyond recalling or reproducing a response

• Items require students to make some decisions as to how to approach the question or problem

• Actions imply more than one mental or cognitive process/step

DOK Level 2: Skills/Concepts

59

Discuss at your table what DOK 2 would look like in a classroom. What would the teacher be doing, what would students be doing?

• Requires deep understanding exhibited through planning, using evidence, and more demanding cognitive reasoning

• The cognitive demands are complex and abstract

• An assessment item that has more than one possible answer and requires students to justify the response would most likely be a Level 3

DOK Level 3: Strategic Thinking

60

Discuss at your table what DOK 3 would look like in a classroom. What would the teacher be doing, what would students be doing?

• Requires high cognitive demand and is very complex

• Students are expected to make connections, relate ideas within the content or among content areas, and select or devise one approach among many alternatives on how the situation can be solved

• Due to the complexity of cognitive demand, DOK 4 often requires an extended period of time

DOK Level 4: Extended Thinking

61

Discuss at your table what DOK 4 would look like in a classroom. What would the teacher be doing, what would students be doing?

The Depth of Knowledge is NOT determined by the verb (Bloom’s Taxonomy), but by the context in which the verb is used and the depth of thinking required. The intended student learning outcome determines the DOK level.

Depth of Knowledge

62

Bloom

What type of thinking is needed

(verbs) to complete the task

Webb

How deeply do you need to

understand the content to be successful?

How Is the Learning Assessed?

Looking at Alignment

63

Adapted from: http://karolyeatts.com/Polk%20Math%20Institute%202008/Webb%20and%20Bloom%20Comparison.pdf

Guide Page

18

• DOK 3: Describe how the author of a short story must be cognizant of how much space he actually has to develop the elements of the story because he is confined by space..(Requires deep understanding of the elements of a short story and definition of the short story genre.)

• DOK 2: Describe the difference between a short story and a novel. (Requires cognitive processing to determine the difference in the two genres.)

• DOK 1: Describe the characteristics of a short story. (Simple recall)

Examining Depth of Knowledge

64

From: Depth of Knowledge: Promoting Rigor and Relevance in Learning. Fulton County School District

Examining Depth of Knowledge in Classrooms1. Before beginning the session, divide into groups specifically for

elementary, middle, and high school. Once in your grade alike groups, break up into smaller groups of 4-6.

2. In your Participant Guide on pages 19-26, read the overview regarding instructional practices at each DOK Level.

3. Individually, for each DOK Level, determine if the task examples are observed during classroom walkthroughs at your charter school. If they are observed, assign a Level of Use score for each task.

4. When all four DOK Levels are completed, in table groups compare your results. Discuss the Reflection Questions on page 27. Note the strategies to help support increased rigor in your school if not currently seen.

5. Reflect on what changes will be needed in your Implementation Plan to increase rigor and Depth of Knowledge.

Activity 5: Examining DOK in Classrooms

65

Guide Pages 19-27

Activity 6: Next Steps

66

Big Ideas People to Share With

Section 1: Teacher Evaluation-The Value-Added Model

Section 2: Understanding Academic Rigor

Section 3: Connecting Academic Rigor and Depth of Knowledge

What are some “big ideas” that you want to remember from today? With whom do you need to share within your charter school in order to take the next steps to support the transition to the Florida Standards?

Guide Page

28

Closing Activities

67

• Developed an understanding of Florida’s Value-Added Model (VAM) and its implications to teachers as well as the entire charter school.

Increased awareness of the higher standards of learning and rigor the new assessments will require of students and staff.

Identified best practices that teachers should be implementing to increase rigor in the classroom.

Developed an understanding of teacher and student behaviors that should be observed during classroom walkthroughs to ensure a rigorous classroom.

Recognized the importance of Webb’s Depth of Knowledge Levels and its connection to increased rigor.

Module Outcomes

68

Post-Assessment and Session Evaluation

69

Where Are You Now?

Assessing Your Learning

Guide Page

30