Evaluation Systems: Context, Issues, and Perspectives

Charles Igel Senior Researcher, McREL

Brian Ewert Superintendent, Englewood Public Schools

Barbara Lunsford Associate Superintendent, Georgia DOE

Kerry Englert President, Seneca Consulting

Jessica Allen Researcher, McREL

Tony Davis Senior Director, McREL

Framework for Evaluation Systems

Teacher Effectiveness

Quality Standards

Leadership Environment Content Learning Practice Growth

50%

50%

Performance Ratings

Developing Proficient Accomplished Distinguished

Professional Practice

Examples of large-scale evaluation initiatives

Considerations for developing and rolling-out educator evaluation systems

Colorado SCEE Example

Human Judgment: Data should inform decisions, but human judgment will always be an essential component of evaluations

Continuous Improvement: The implementation and assessment of the evaluation system must embody continuous improvement

Feedback: The purpose of the system is to provide meaningful and credible feedback that improves performance

Alignment: Educator evaluation must take place within a larger system that is aligned and supportive

State Council for Educator Effectiveness. (2011, May). State Council for Educator Effectiveness report and recommendations. Denver, CO. Available at www.cde.state.co.us/EducatorEffectiveness

Additional considerations: Educational Leadership Policy Standards (ISLLC 2008)

S1: Facilitating the development and implementation of a shared vision for learning

S2: Nurturing a culture conducive to student learning and professional growth

S3: Managing for a safe, efficient, and effective learning environment

S4: Collaborating with and mobilizing faculty and community resources

S5: Acting in a fair and ethical mannerS6: Responding to and influencing the political, legal, and cultural

environment

Additional considerations: InTASC Model Core Teaching StandardsS1: Uses of developmentally appropriate instructionS2: Establishes inclusive learning environmentsS3: Promotes positive peer interactions and engagement with materialS4: Understands content and makes it meaningful and accessible to all

learnersS5: Connects content in ways that promote creativity and critical

thinkingS6: Uses multiple methods of assessmentS7: Uses all professional knowledge in lesson planningS8: Uses multiple instructional strategies S9: Active in own professional growthS10: Take appropriate leadership positions and actively engages in the

profession

Additional considerations: Educational Data Exchange Network (EDEN) Reporting

What is reported?1) Teacher counts categorized by performance level 2) Total number of teacher evaluated3) Principal counts categorized by performance level4) Total number of principals evaluated

Guidance on –1) teachers and/or principals assigned to multiple schools in the same LEA2) teachers and/or principals assigned to multiple schools in different LEAs

Charles Igel, PhDMid-continent Research for

Education & Learning (McREL)cigel@mcrel.org

303.632.5574

THE Sinking of THE Sinking of the Titanicthe Titanic

Purpose of EvaluationPurpose of Evaluation• Serves as an accountability and compliance system

• Serves as a tool to help in employment decisions

• Provides a set of rigorous research-based standards

• Serves as a measurement of performance and effectiveness

• Promotes leadership

• Serves as a personal, reflective tool to become more effective

• Serves as the basis of instructional improvement (and school improvement)

• Enhances the implementation of the approved curriculum

• Guides/drives continuous improvement through professional development

• Serves as as a tool in a coaching/mentoring model

• Serves as a tool to determine performance pay (in some places)

• Informs higher education & research

Englewood Schools: Englewood Schools: ICEBERG DEAD AHEAD!ICEBERG DEAD AHEAD!

• Lack of Expectations & Nonnegotiables

• Lack of Accountability

• Lack of Transparency

• Lack of Quality

• Lack of Fidelity

• Lack of Consistency

• Lack of Reliability & Validity

• Lack of Integrity

ICEBERG DEAD AHEAD!ICEBERG DEAD AHEAD!

• Lack of Reflective Practice

• Deficit/Punitive Model vs. Continuous Improvement Model

• Lack of Performance Pathways for Growth

• Lack of Policies, Procedures, and Agreed Upon Practices

• Lack of a Framework to Drive Conversation

• Lack of Goal Setting and Professional Growth

The LifeboatThe Lifeboat

• Teacher/Principal Effectiveness Bill

• Creates a four-tiered system to evaluate the effectiveness of licensed personnel as a means of improving the quality of education in Colorado. The basic purposes of the statewide system are:

To ensure that all licensed personnel are evaluated using multiple, fair, transparent, timely, rigorous, and valid methods, fifty percent of which evaluation is determined by the academic growth of their students

To ensure that all licensed personnel receive adequate feedback and professional development support to provide them a meaningful opportunity to improve their effectiveness

To ensure that all licensed personnel are provided the means to share effective practices with other educators throughout the state

The LifeboatThe Lifeboat

• Solidarity & Alignment: Board, Administrators, Principals, & Union

• Non-negotiables and Expectations

• Accountability: 100% Participation

• Coaching/Collaborative/Reflective Model

• Clearly Defined Framework with Common Language

• Low Stakes Implementation with: Quality, Fidelity, Intensity, Consistency

The LIfeboatThe LIfeboat

• 21st Century Instructional Model

• State and Common Core Standards

• Data Teams

• Tied to School & District Improvement Plans

• Performance Pathways for Growth

• Aligned, Collaborative Goal Setting

• Professional Development Based on Data

RMS Carpathia &RMS Carpathia &Pier 54-NYCPier 54-NYC

• Much Work To Do!

• Robust, Reliable, Intuitive, Flexible, & Easy-to-Use Technology & Web Tools

• Seamless Integration with Other Initiatives

• 8:1 Ratio

• Peer Coaching and Evaluation

• Quality-Fidelity-Intensity-Consistency-Reliability-Validity

RMS Carpathia &RMS Carpathia &Pier 54-NYCPier 54-NYC

• Formative and Summative Observation & Evaluation Data

• Deep, Meaningful Conversations around Teacher Standards, Elements, Rubrics, & Ratings

• Real Time, Needs-Based Professional Development

• Value-Added, Student Growth Measures

• Valid, Reliable,Vetted Metrics & Algorithms

• Group/Collective Performance Pay

• Employment Decisions

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

Georgia Department of Education

6/19/2011 19

CLASS KeysClassroom Analysis of State Standards

A Teacher Evaluation Process Standards

Barbara LunsfordAssociate Superintendent

School Improvement

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

CLASS KeysSM Strands and Elements

6/19/2011 20

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

ASSESSMENT - The collecting and analyzing of student performance data to identify patterns of achievement and underachievement in order to design and implement appropriate instructional interventions.

STRAND STANDARD ELEMENT

CONTINUUM OF IMPROVEMENT RUBRIC

6/19/2011 21

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

Not Evident Emerging Proficient Exemplary

6/19/2011 22

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org 6/19/2011 23

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

Performance on the elements of the CLASS KeysSM is identified on a four-level continuum.

Not EvidentNot Evident EmergingEmerging ProficientProficient ExemplaryExemplary

This continuum is not utilized to label teachers as Not Evident, Emerging,

Proficient, or Exemplary.

The continuum is used to describe a teacher’s PERFORMANCE on

specific elements.

6/19/2011 24

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

GTDRPerformance

An array of evidence is collected from multiple sources during the year.

AnnualAnnualEvaluationEvaluation

Announced,Longer

Observations

Unannounced, Short

Observations

Other artifacts and evidence from

conferences, meetings, review

of teacher and student productsStudent Achievement Data

6/19/2011 25

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

Module 1: Content and Structure (The What?)

Module 2: Overview of the Evaluation Process (The How?)

Module 3: Self-Assessment and Reflection

Module 4: Professional Growth Plan

Module 5: Pre-Evaluation Conference

Module 6: Informal Observations

Module 7: Formal Observations

Module 8: Georgia Teacher Duties and Responsibilities

Module 9: Annual Evaluation

Module 10: Professional Development Plan

CLASS KeysSM Modules

CLASS KeysSM

Modules

6/19/2011 26

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org

Evaluator Training Assessment

Three Sections:• Scoring the Elements in a Classroom Video• Annual Evaluation Performance Task• Technical Questions Regarding Process

6/19/2011 27

Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org 6/19/2011 28

KERRY ENGLERT, PH.D.SENECA CONSULTING

JESSICA ALLEN, MAMcREL

COUNCIL OF CHIEF STATE SCHOOL OFFICERSNATIONAL CONFERENCE ON STUDENT ASSESSMENT

JUNE 19TH, 2011

Validity in a Standards-Based Teacher Evaluation

System

Presentation Purposes

Overview of our experiences conducting an initial validity study of a state’s standards-based teacher evaluation system. Propose a framework that can be used when

designing these validation studies. Offer suggestions and considerations for validation

studies of educator evaluation systems.

National Issues in Teacher Effectiveness

Education Research Connection between teacher effectiveness and student

learning that is well documented (ex. Goe (2007) ).

Education Policy Focusing on teacher evaluation and the relationship

between teacher effectiveness and student achievement. Race to the Top White House’s Blue Print for Reform

Standards-Based Evaluation System

Three core components: A set of standards. Procedures for collecting performance data on the

standards. Rubrics to evaluate performance.

Optional fourth component: Methodology for incorporating evaluation results into

personnel decisions.

* from (Odden, 2004):

Why do a validity study?

Race to the Top Requires a valid and reliable teacher evaluation

instruments.

Validity evidence Builds credibility to the instrument and evaluation

system. Provides information on data use and implementation

Provides information on continual development. Can focus training efforts.

CLASS KeysSM Teacher Evaluation System

Purposes That serves a two-fold purpose improvement and

accountability

Core Components Five major strands Training materials 28 performance rubrics.

Validity Study of CLASS KeysSM System

Study Context Quick turn around time Extant data from pilot study Readily available and accessible information

Broad examination of the literature Inform, focus, and guide the study Better capitalize on the available data

Relevant Literature

Validity Theory and Standards A unified theory and system uses (Standards for

Educational and Psychological Testing, 1999; Messick, 1989)

A strong theory of action (Kane, 2006) Personnel Evaluation 

Applying the concepts of fairness, transparency, efficiency and practicality to the personnel evaluation context (Gullickson, 2008)

Quality Rubrics Characteristics of quality rubrics (Arter & McTighe, 2001)

Implementation Fidelity The degree to which the system is being implemented

according to the specification (Ruiz-Primo, 2006)

Potential framework for validity in Educator Evaluation

Validity in

Educator

Evaluation

Methods related to the framework

Validity Theory and Rubric Quality Content

Validity Theory Construct

Personnel Evaluation Standards and Validity Theory Fairness and Reliability

Implementation Fidelity and Validity Theory Use and Interpretation

Future Considerations

Validation of educator evaluation systems might benefit from multiple perspectives.

Pilot and field tests are opportunities not only to test the instrument but apply validity methodologies and instruments.

Validation is an ongoing process and provides a framework to understand how the instrument is being used.

References

American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association.

Arter, J., & McTighe, J. (2001). Scoring rubrics in the classroom: Using performance criteria for assessing and improving student performance. Thousand Oaks, CA: Corwin.

Goe, L. (2007). The link between teacher quality and student outcomes: A research synthesis. National Comprehensive Center for Teacher Quality. Retrieved from http://www.tqsource.org/publications/LinkBetweenTQandStudentOutcomes.pdf

Gullickson, A. R. (2008). The Personnel Evaluation Standards: How to Assess Systems for Evaluating Educators, 2nd edition. Joint Committee on the Standards for Educational Evaluation. Corwin Press, Thousand Oakes, CA.

Kane, M. T. (2006). Validity. In R. L. Brennen (Ed.), Educational Measurement (Vol. 4th edition). Westport, CT: Praeger Publishers.

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13-103). New York: American Council on Education & McMillian.

Odden, A. (2004). Lessons learned about standards-based teacher evaluation systems. Peabody Journal of Education, 79(4), 126-137.

Ruiz-Primo, M. A. (2006). A multi-method and multi-source approach for studying fidelity of implementation. National Center for Research on Evaluation, Standards, and Student Testing (CRESST).