September 23 , 2013 12:30 pm – 1:45 pm

S.B. 10-191 Implementation:

Implementing the Colorado Model System

for Measures of Student

Learning September 23, 2013 12:30 pm – 1:45 pm

Before we begin…

Find your work group – it ’s important where you sit. Try to sit with members of your school or district.

High Knowledge

Low Knowledge

NextHigh ComfortLow Comfort

On the wall you will see a chart that will be used to capture everyone’s level of comfort and knowledge with Measures of Student Learning. Place a dot on the chart the best represents your level of comfort and knowledge currently .

VisionAll students in Colorado will become educated and

productive citizens capable of succeeding in a globally competitive workforce.

MissionThe mission of CDE is to shape, support, and safeguard a statewide education system that prepares all students for

success in a globally competitive world.

Together We Can

Successful studentsPrepare students to thrive in their education and in a globally competitive workforce. Ensure every student is on track to graduate postsecondary and workforce ready. Increase achievement for all students and close achievement gaps. Ensure students graduate ready for success in postsecondary education and the workforce. Increase national and international competitiveness for all students.

Great teachers and leadersEnsure effective educators for every student and effective leaders for every school and district. Increase and support the effectiveness of all educators. Optimize the preparation, retention, and effectiveness of new educators. Eliminate the educator equity gap.

Outstanding schools and districtsBuild the capacity of schools and districts to meet the needs of Colorado students and their families. Increase school and district performance. Turnaround the state’s lowest performing districts and schools. Foster innovation and expand access to a rich array of high quality school choices for students.

Best education system in the nationBuild the best education system in the nation. Lead the nation in policy, innovation, and positive outcomes for students. Operate with excellence, efficiency, and effectiveness to become the best SEA in the nation. Attract and retain outstanding talent to CDE.

Goals

Students

Educators

Schools/ Districts

State

Driving Questions

What do we want students, educators, schools, and districts to know and be able to do?

How will we know if expectations are met?

How will we respond when help is needed and to support continued growth?

Colorado Academic Standards Assessments

• RTI• PBIS• Targeted interventions• IEPs

Educator Quality Standards

Educator evaluations

• Induction• Mentoring• Professional development plans• Remediation plans

Performance Indicators

School and district performance frameworks

• Unified planning• Priority• Turnaround

Students

Educators

Schools/Districts

When we intenti onally integrate… Coherent and rigorous academic standards

Innovative and engaging learning options

Supported and effective educators

Aligned and meaningful assessments

Statewide and district accountability

Expanding Student Learning

We can personalize learning and ignite the potential of every student.

Introductions

How many of you have attended a CDE training or a training in your

school/district on Measures of

Student Learning?

Welcome and Purpose IntroductionsMeasures of Student Learning

Steps 1-5 Create a template/pie chart

Feedback and critical questioning Decision Framework

Work Session and Next Steps Closing – Feedback and Reflection

Agenda Measures of Student Learning

Training Objectives: Desired Outcomes

Why am I here?To understand the shift from student

growth to measures of student learning To gain a clear understanding of the 5

steps for including measures of student learning in your evaluation system

To determine next steps for your school, district or BOCES

Meeting Norms: How We Will Play Together

How will we work together… Honor everyone’s ti me Listen to learn and apply to your context Balance parti cipati on and share airspace Paraphrase, probe, pause Suspend judgment Commit to practi ce and follow-through Avoid technological distracti ons Have a sense of humor

Measures of Student Learning

STATE COUNCIL FOR EDUCATOR EFFECTIVENESS

Framework for System to Evaluate Teachers

Definition of Teacher Effectiveness

I. Know Content

50% Professional Practice Standards 50% Student Growth MeasuresWeighting: How Much Does

Each Standard Count Towards Overall Performance?

Observations of Other Measures Teaching Aligned with

CDE Guidelines

State Other Assessments Other MeasuresSummative for Non-tested Aligned with Assessments Areas CDE Guidelines

Match of test to teaching assignments

Weighting:Scoring Framework: How Do Measures of Quality Standards

Result in a Determination of Individual Performance?

Performance RatingsIneffective Partially Effective Effective Highly Effective

Quality StandardsII. Establish

EnvironmentIII. Facilitate

LearningIV. Reflect on

PracticeV. Demonstrate

LeadershipVI. Student

Growth

Appeals Process

From Student Academic Growthto


Measures of Student Learning Using multiple measures to

determine student learning over time.

Colorado Growth Model (CGM)

establishes technical

measure of “growth.”

BrainstormHow do you measure student learning in your

district/school?


Approaches for Selecting and Using Multiple Measures in Educator

Evaluation

Step 1: Review of the assessment requirements

Step 3: Select and weight multiple measures of student learning

Step 2: Determine student learning measures in your district

Step 4: Set student learning outcome targets and scales

Step 5: Combine the results to get a single student outcomes rating

Decision Framework

Teacher Evaluations

50% Student Academic Growth

VI. Responsibility for student academic growth

Evaluated using the following:(1) a measure of individually-attributed growth, (2) a measure of collectively-attributed growth; (3) when available, statewide summative assessment results; and (4) for subjects with statewide summative assessment results available in two consecutive grades, results from the Colorado Growth Model.

Refers to outcomes on a measure that are

attributed to an individual licensed person,

e.g. DRA2 outcomes for a 1st Grade Teacher’s

students

Refers to outcomes on a measure attributed to two or more licensed

personnel, e.g. 10th gr. Math TCAP –

All Secondary math teachers in school

Table Talk:At your table, discuss if you want more

collective or individual attribution in your evaluation system. Consider the advantages and unintended

consequences of both collective and individual. Be prepared to share your thinking.

Step 1: Collective vs. Individual Attribution


Evaluation






Decision Framework

Determine how student learning is currently measured in your districtConduct an assessment inventory to identify

what is currently being used to measure student learning

Identify where gaps exist

Step 2: Measures of Student Learning

A Step-By-Step Guide for Selecting and Using Student Learning

Outcomes






Decision Framework

What do we value? How might we categorize our teachers?

How will our decisions reflect our values?

Categorizing Teachers

No Categorization

Categorization

Group Decisions

Individual Decisions

Example:

Categorizing Teachers

No Summative/CGMAll Other Teachers

Colorado Growth Model (CGM)

Reading, Writing, Math (Gr. 4-10)

2.

State SummativeReading, Writing, Math

(Gr. 3)

Science (Gr. 5, 8, 10)

Social Studies (Gr. 4, 7, HS)

ACT (HS)WIDA ACCESS (ELL)

What are the pros and cons of creating groups of like teachers based on the types of measures that must be included in their body of evidence?

Based on what is expected in law, what categories of teachers are possible?

Select and weight multiple measures of student learning to be included in educator evaluations.What can we do as a district to build comparable

bodies of evidence in like groups of teachers?Let’s try it!

Measures of Student Learning Tool


Create a template using the Measures of Student Learning tool. Questi ons to consider:

What assessments must we include?How much influence will they have on the evaluation?

Create a pie chart for a category of teacher or individual teacher that shows your thinking.

Include a rationale.

Creating Specific Weighting Templates

Pie Chart Feedback

Round RobinEach group should select a spokesperson to

share your thinking and rationale for your choices.

Audience should consider feedback on:The collective vs. the individual attributionTypes of assessmentsWeights

Debrief/Reflection

Did the feedback help shape any of your thinking?

Any aha’s?

What might you reconsider/change if anything?

Districts, BOCES and schools may decide aft er completi ng the Assessment Inventory that a locally created assessment would likely be included as a measure in evaluati on.

Districts and BOCES can use the Assessment Review Tool to show they are taking steps to ensure that the locally created measurements they select are fair, valid, and reliable.

Using Local Assessments in Evaluation

Assessment Review Tool

Criteria used in this tool:Alignment Scoring Fair and UnbiasedOpportunities to Learn

Assessment Review Tool

Assessment Support

Content Collaborati ves P-12 educators from around the state gathered to identi fy and

create a high-quality assessment resource bank, which is aligned to the new Colorado Academic Standards and may be used in the context of Educator Eff ecti veness evaluati ons.

The Content Collaborati ves, CDE, along with state and nati onal experts, will establish examples of student learning measures within each K – 12 content area including:

Cohort I

Dance Drama & Theatre Arts Music Reading, Writing and Communicating

Social Studies Visual Arts

Cohort IIPhysical Education Science World Languages Comprehensive Health

Mathematics CTE

Decision Points How will our district create templates for different

teacher roles and the types of assessments that exist?Who will participate in the process? What do we need to know in order to make good

decisions?What resources (internal and external) do we

have/need?How will we know if we have made good decisions?

Categorizing, Weighting, and Selecting Measures Reflection






Decision Framework


Evaluation

Use the Student Learning Objective Process to set outcome targets and scales based on data from the selected measures.


What is the Student Learning Objective Process? Enables educators to utilize academic standards

to establish learning outcomes for students, monitor progress and evaluate the degree to which students achieve those outcomes

In educator evaluation systems, this process serves as a: Method of measuring student growth, progress or

mastery of the standards

Student Learning Objective Process

The essential steps of the Student Learning Objective Process include: Understanding the Colorado Academic Standards in order to

select learning outcomes Collecting baseline information to inform target and scale

setti ng Assessing quality, attainment level and rigor of student learning

targets and scales Monitoring student learning (formative practice) Determining attainment of student learning targets and scales Reflecting and refining the Student Learning Objective Process



Turn and Talk:• Why is it important to start the Student Learning

Objective process with the Colorado Academic Standards?

• How does this first step support the work done in professional practice?

• What tool is in place to support you with part of this process?

THE PROCESS OF: Understanding the Colorado Academic Standards

THE PROCESS OF: Selecting Learning Outcomes for Target

Setting

THE PROCESS OF: Procuring or Developing Assessments Using

Quality Criteria


THE PROCESS OF: Collecting Baseline Information

THE PROCESS OF: Setting Student Learning Targets

THE PROCESS OF: Setting Appropriate Scales for Measurement

THE PROCESS OF: Understanding the Colorado Academic Standards

THE PROCESS OF: Selecting Learning Outcomes for Target Setting

THE PROCESS OF: Procuring or Developing Assessments Using Quality Criteria

Possible Student Learning Targets and Scales

Examples of Student Learning Targets and Scales:16 of my 21 students will increase their reading

proficiency by 1-2 grade levels. 9 more students will receive + at the end of the

year. The median score on my end of course

assessment last year was 73%. At the end of the course for this year, the median score will be at least 78%.

Student Learning Objective Process:

Strategies for Target Setting Possible strategies to consider when determining how

students will be captured when target setti ng: Individual – Teacher sets individual goals for students and then

determines how many students met their individual goals Whole Class – Teacher sets a goal that would be applied to all

students in a class Subgroup of Students – Teacher sets a goal for a group(s) of

students with similar baseline data levels Case Load – Teacher sets a goal for the specific students he or she

supports

Some of these strategies may work bett er with some teaching assignments. Can you think of which teaching assignments might

work best with which strategy based on your local context?


Strategies for Target Setting Possible strategies to consider when setti ng learning targets for

students: All targets are indicators of growth using a variety of data sources: Proficiency data Targets – students meeting grade level

expectations Examples: a % of students will pass the end-of-course exam; a % of

students will score a 3 or bett er on the AP exam Growth data Targets – students growing over the course of

instruction Examples: a % of students will progress one fi tness level; a % of

students will make projected growth or bett er on the MAP assessment Averaging data Targets – students’ average score on an

assessment Example: Students will answer, on average, 80% of the questions

correctly on the end-of-course exam

Possible Student Learning Targets

What are the targets in these examples?16 of my 21 students will increase their reading

proficiency by 1-2 grade levels. 9 more students will receive + at the end of the

year. The median score on my end of course

assessment last year was 73%. At the end of the course for this year, the median score will be at least 78%.


Strategies for Scale SettingTeachers set ranges (how many students are

expected to meet the set target) across 4 rating levels: above expected, expected, less than expected, and much less than expected

Teachers collaborati vely set ranges with their evaluator Ranges will vary from year to year and possibly class to

class as baseline data will vary due to student beginning points

Collaboratively setti ng ranges will allow teachers and evaluators to calibrate about expected student learning outcomes

You can use the Measures of Student Learning Tool to record the targets and scales you have established.

When Colorado Growth Model data is available the scale indicating expected student learning outcomes has been established for you.


Strategies for Scale Setting

Possible Student Learning Scales

What are the scales in these examples?16 of my 21 students will increase their reading

proficiency by 1-2 grade levels. 9 more students will receive + at the end of the year. The median score on my end of course assessment

last year was 73%. At the end of the course for this year, the median score will be at least 78%.

Student Learning Objective Process: Target Setting

PracticeWith a partner, use the Setti ng Student Learning

Targets and Scales worksheet to analyze one set of data.Portfolio/Performance OutcomesBaseline Reading LevelsEnd of Course Summative Exam ResultsRubric Outcomes

Practice setti ng a student learning outcome target for expected outcomes.


Scale Setting Practice Think about what your expected outcomes are

(targets). Establish the ranges for each of the categories

below.

How did you decide on the ranges? Are your expectations rigorous, yet att ainable?

ALL 4 sets of dataPerformance Assessment/Portfolio Outcomes

Reading Levels (beginning of the year)

Cumulative exam

Rubric Results


Target and Scale Setting

66 89 68 75 74 80 94 99 60 55 71 73 95 96 84 82 79 81 98 74 86

6 4 1 5 5 2 6 2 5 2 1 6 3 3 5 3 1 2 6 4 4

18 24 24 30 38 40 40 50 50 50 50 50 60 60 60 60 70 70 70 80 80

+ - + - + - + - - + + -


THE PROCESS OF: Assessing Quality, Attainment Level and Rigor of Student Learning Targets and Scales

THE PROCESS OF: Collecting Baseline Information THE PROCESS OF: Setting Student Learning

Targets THE PROCESS OF: Setting Appropriate Scales for

Measurement

THE PROCESS OF: Understanding the Colorado Academic Standards THE PROCESS OF: Selecting Learning Outcomes

for Target Setting THE PROCESS OF: Procuring or Developing

Assessments Using Quality Criteria


Determine Target and Scale Quality Will your district establish quality criteria for

the use of Student Learning Targets and Scales?Criteria for establishing quality targets and

scales might include:Approval processesCollaboratively establishedLevel of attainmentRigor Based on previous dataMeet or exceed past student performance


THE PROCESS OF: Monitoring Student Learning (Formative Practice)


THE PROCESS OF: Collecting Baseline Information THE PROCESS OF: Setting Student Learning Targets THE PROCESS OF: Setting Appropriate Scales for Measurement

THE PROCESS OF: Understanding the Colorado Academic Standards THE PROCESS OF: Selecting Learning Outcomes for Target Setting THE PROCESS OF: Procuring or Developing Assessments Using Quality

Criteria


THE PROCESS OF: Determining Attainment of Student Learning Targets and Scales





Criteria

Performance Assessment/Portfolio Outcomes

Reading Levels (beginning of the year)

End of year cumulative exam

Rubric results

Student Learning Targets:Reflection on Pre and Post

Data

75 90 76 89 80 99 90 91 80 70 82 79 94 95 98 95 66 87 98 85 71

5 5 3 6 6 4 3 5 6 6 4 5 4 4 6 5 3 6 5 6 5

30 40 38 38 50 50 50 60 50 60 70 70 60 70 70 80 80 70 80 80 80

+ + - + + + + ++

- + +


THE PROCESS OF: Reflecting and Refining Student Learning Objective Process

THE PROCESS OF: Determining Attainment of Student Learning Targets and Scales





Criteria

Considerations when including the Student Learning Objective Process in evaluation:Assessment literacy of educators creating targetsData literacy of educators Time and effort required of teacher and evaluator Focus of school or district (all students and/or subgroups of

students) Available information or data in school or district Ease or readiness level of teachers and evaluators Possible approval process of strategies selected Number of targets wanted as a district/building

Considerations when Setting Student Learning Targets

ReflectionWhat are the pros and cons of using the Student

Learning Objective Process?What are teachers already doing to support process?

Are there groups of teachers that already do something very similar to creating Student Learning Targets and Scales?

What additional ideas do you have for making the Student Learning Objective Process more manageable?






Step 5: Combine the results to get a single student learning outcomes rating

Decision Framework


Evaluation

Combine the results of multiple measures to get a single measures of student learning rating.


Draft Example

Combining Outcomes

Putti ng It All Together The Decision Framework helps districts

aggregate the 50% Professional Practices and the 50% Measures of Student Learning to reach a final rating of one of the following: Highly EffectiveEffectivePartially Effective Ineffective

Decision Framework

Not Evident

Partially Proficient

ProficientAccom-plished

Exem-plary

1

1

2

2

4

0

20.0% 108 * 10 / 24 = 45.03

4

2

1

1

1

20.0% 108 * 12 / 24 = 54.04

3

2

1

2

2

0

1

30.0% 162 * 15 / 32 = 75.9

1

1

2

15.0% 81 * 4 / 12 = 27.0

2

1

1

4

15.0% 81 * 8 / 16 = 40.5

100% 540 242.4Pts earned =

Weights by std

Points Possible

* earned / possible =

Points Earned

f. Creates a well managed learning environment

Overall Rating for Quality Standard II

Rating

Proficient

Proficient

a. Provides aligned instruction

b. Demonstrates knowledge of literacy development

c. Demonstrates knowledge of mathematics (All)

d. Demonstrates knowledge of content

e. Develops interconnected lessons

f. Makes instruction and content relevant to students

Overall Rating for Quality Standard I

II. Learning Environment

a. Fosters a predictable, caring learning environment

b. Demonstrates a commitment and respect for diversity

c. Engages students

III. Instruction

a. Demonstrates knowledge of current developmental science

b. Plans and delivers data driven instruction

c. Demonstrates knowledge of effective instructional practices

d. Integrates and utilizes technology

e. Establishes high expectations for students

f. Provides opportunities to develop leadership qualities

g. Communicates to students effectively (objectives)

h. Uses appropriate methods to assess

Overall Rating for Quality Standard III

d. Adapts teaching to meet individual needs

e. Works and communicates with families

Proficient

V. Leadership

Partially Proficient

ProficientOverall Rating

a. Demonstrates leadershipb. Contributes to the teaching professionc. Advocates for schools and studentsd. Demonstrates high ethical standards

Overall Rating for Quality Standard V

IV. Reflection

a. Analyzes student data and applies to instruction

b. Links professional growth to professional goals

c. Is able to respond to a complex, dynamic environment

Overall Rating for Quality Standard IV

Proficient

Quality Standard

Element

I. Content and

Pedagogical Knowledge

Prof

essi

onal

Pra

ctice

s Re

port

How to combine professional practice and measures of student learning ratings


Prof

essi

onal

Pra

ctice

s

540

Exemplary(433 to 540 pts)

Accomplished(325 to 432 pts)

Proficient(217 to 324 pts)

Partially Proficient(109to 216pts)

Basic(0 to 108pts)

Much Lower Than Expected Growth

(0 to 134 pts)

Lower Than Expected Growth(135 to 269 pts)

Expected Growth(270 to 404 pts)

Higher Than Expected Growth(405 to 540 pts)

Highly Effective

730-1080Effective

487-729Partially Effective

244-486Ineffective0-243

0

0

540

2. Annual

Orientation

3.Self-

Assessment

4.Review of

Annual Goals and

Performance Plan

5.Mid-Year Review

6.Evaluator

Assessment

7.End-of-Year

Review

8.Final Ratings

9.Goal-Setting

and Performance

Planning

1.Training

1.Training

Evaluation CyclePrincipal/AssistantPrincipals and Teachers

May 15Within the first two weeks of

school.

End of September.

Prior to the beginning of

Spring Semester

Train: Prior to the beginning of

School.Orient: Within

the first week of School.

End of MayMid-JuneEnd of June

Review and finalize compiled results

from measures of student learning

Include any new district or school

decisions for measuring student

learning

Confirm measures used to determine student learning– Finalize baseline

data

Review available data from measures

to determine if students are on

track

Consider decision points, priorities and what you value.

Identify next steps for your Performance Evaluation Committ ee.

What are your next steps?

Let’s Get to Work!

CDE Educator Eff ecti veness website: htt p://www.cde.state.co.us/EducatorEff ecti veness/ FAQ’s, Fact Sheets Policy and State Model Evaluation System Info Six Steps guidance document

http://www.cde.state.co.us/EducatorEffectiveness/StudentGrowthGuide.asp

CDE Educator Eff ecti veness e-newslett erhttp://www.cde.state.co.us/scripts/communications/EE

Newsletter.aspStay informed by signing up for this monthly newsletter

http://www.cde.state.co.us/EducatorEffectiveness/




http://www.cde.state.co.us/scripts/communications/EENewsletter.asp

http://www.cde.state.co.us/scripts/communications/EENewsletter.asp

Where are you now?

High Knowledge

Low Knowledge

Next

High ComfortLow Comfort

On the wall you will see a chart that will be used to capture everyone’s level of comfort and knowledge with Measures of

Student Learning. Place a dot on the chart the best represents your level of comfort and knowledge now that we are near the

end of the training.

Feedback

Please take a few minutes to complete your feedback form.

Contact Us

EE Leadership Katy Anthes: Executive Director

[email protected] Toby King: Director

[email protected] Jean Williams: Rubric Evaluation

Specialist [email protected]

Colorado Legacy Foundation Mike Gradoz: Director

[email protected]

Communications Amy Skinner

[email protected] Katie Lams:

[email protected]

Britt Wilkenfeld: Data Fellow [email protected]

Tricia Majors: Project Mgr. [email protected]

mailto:[email protected]








Implementati on Support and Development

Courtney Cabrera [email protected]

Sed Keller [email protected]

Dawn Pare [email protected]

Bob Snead [email protected]

Chris Vance [email protected]

Contact Us

Colorado Department of Higher Education Jennifer Arzberger

[email protected]







September 23 , 2013 12:30 pm – 1:45 pm

Documents

Transcript of September 23 , 2013 12:30 pm – 1:45 pm