Features of EAR Classroom Visit Protocol - Yola€¦ · IRRE, 2010. 1 Measuring What Matters© EAR...

IRRE, 2010. 1

Measuring What Matters© EAR Classroom Visit Protocol Overview and Description of Instrument, Reports, Technical Supports, and Training Activities

Institute for Research and Reform in Education February, 2010

IRRE, 2010. 2

MEASURING WHAT MATTERS©

AT A GLANCE

Measuring What Matters (MWM) arms our partner districts and schools with the data they need to

measure the outcomes and implementation of their reform.

MWM encompasses:

A set of data collection tools and reports focused on student outcomes and effective practices.

A process for using these tools and reports effectively.

MWM student outcome reports developed by our partner districts with assistance from MWM staff are

made available to:

Family advocates so they can monitor the progress of individual students and share that

information in family conferences and one-on-one discussions with students.

Coordinators and staff in small learning communities (SLC) so they can plan and monitor

interventions and supports.

School and district leaders so they can track and report progress; design interventions; and assess

the system’s effectiveness in hitting outcome targets.

Community stakeholders so they celebrate success and hold the system accountable for student

progress.

MWM also includes data collection, analysis and reporting tools developed by MWM staff and consultants

around effective practices to support each of FTF’s three implementation strategies: small learning

communities, a Family and Student Advocate System and instructional improvement. Teachers,

administrators and technical assistance providers can use these tools to collect data through observation,

self-assessments and surveys. The data then become a solid basis for shaping supports for individual staff

as well as more systemic interventions at the departmental, SLC, school, and district levels.

IRRE provides intensive training and ongoing supports to district and school staff in the use of the MWM

tools, including:

IRRE, 2010. 3

What MWM measures and why.

How to collect data reliably using these tools.

How to interpret and use results to strengthen specific components of the reform.

MWM’s tools and training provide schools and districts the capacity to sustain and strengthen their reform

efforts using a small number of compelling success indicators.

The following pages provide a detailed summary of MWM and include:

An overview of Effective Practices (EP) measurement

descriptions of the effective practices targeted by MWM with conceptual definitions and supporting research

description of the psychometric properties of the EAR Classroom Visit Protocol©

descriptions of the observation protocols, surveys and self-assessments available through MWM that provide measures of the implementation of effective practices

description of the EAR Classroom Visit Protocol©, a generic version of the protocol and screen shots of the protocol as deployed on hand-head devices

a sample of MWM reports available to our partners through our secure on-line MWM website

draft versions of observation protocols in development that measure the implementation of effective advocacy periods and common planning time

a summary of the training and supports offered for users of Effective Practices tools

frequently asked questions about the EAR Classroom Visit Protocol©

Additional information and descriptions of other trainings and technical support are available from IRRE.

© IRRE, 2010. 4

OVERVIEW OF EFFECTIVE PRACTICES MEASUREMENT

MEASURING WHAT MATTERS© Institute for Research and Reform in Education

Measuring What Matters includes data collection, analysis and reporting tools developed by IRRE

staff and consultants around effective practices to support each of FTF’s three implementation

strategies: smaller learning communities, a Student and Family Advocate System and instructional

improvement. Teachers, administrators and technical assistance providers can use these tools to

collect data through observation, self-assessments and surveys. The data then become a solid basis

for shaping supports for individual staff as well as more systemic interventions at the departmental,

SLC, school, and district levels.

IRRE provides intensive training and ongoing supports to district and school staff in the use of the

MWM tools, including:

What MWM measures and why.

How to collect data reliably using these tools.

How to interpret and use results to strengthen specific components of the reform.

Effective Practices tools and training provide schools and districts the capacity to sustain and

strengthen their reform efforts using a small number of indicators of quality and progress.

The following sections describe indicators of effective practices; protocols, surveys and self-

assessments used by instructional leaders, TA providers, and teachers to continually monitor

implementation of reform strategies; as well as a sample of the reports that support data-driven

improvement. We highlight our most mature product, the EAR Classroom Visit Protocol, and describe

the training and technical supports provided by MWM staff to users of Effective Practices tools and

conclude the section with a Frequently Asked Questions document.

© IRRE, 2010. 5

PSYCHOMETRIC EVIDENCE OF THE EAR CLASSROOM VISIT PROTOCOL© Institute for Research and Reform in Education

Reliability

Raters of varying levels of experience conducting structured classroom visits have been found to

make consistent ratings and to agree with other raters using the EAR Classroom Visit Protocol. Both

public school educators and independent external raters have demonstrated that trained observers

can see classroom practice in the same way and with the same metric for quality as do experienced

expert observers of classroom practice. Several studies, both independently sponsored and internally

conducted have found compelling evidence for the use of the EAR Classroom Visit Protocol as a tool

that practitioners and researchers alike can use to measure and monitor the quality of teaching and

learning in public school classrooms.

Trained educators have agreed with each other about the quality of classroom instruction when

observing for a) student engagement, b) alignment of instructional strategies and materials with

curriculum benchmarks and standards, and c) rigor of instructional activities and student work

products. Across multiple districts in Kansas, Missouri, Texas, Mississippi, Florida, California, Arizona

and elsewhere instructional leaders have been trained to criterion and have reached agreement with

their colleagues and with expert raters on the quality of teaching and learning in classrooms. In eight

districts highlighted below instructional leaders participated in training sessions and then conducted

ten classroom visits as teams. Pair-wise agreements among all possible pairs of observers are

presented in Table 1. These newly trained instructional leaders agreed with each other 80% of the

time when observing for student engagement and alignment of instructional strategies and materials

with curricular and state-level standards and benchmarks. More than 90% of the decisions that these

observers made were in agreement about the levels of rigorous teaching and learning.

Table 1. Agreement on EAR Protocol Indicators among Trained Instructional Leaders after Conducting Ten Classroom Visits Together

DISTRICT

Engagement Alignment Rigor

N % Agreement N % Agreement N % Agreement

1 879 81 568 86 489 89

2 458 77 407 81 238 92

3 985 78 985 72 985 89

4 246 84 224 87 181 93

5 127 83 128 68 127 96 6 1,128 81 898 83 703 94

7 66 89 69 88 74 94

8 67 83 67 92 67 90

Total 3,956 80.2 3,346 80.0 2,864 91.2

Notes: N = Number of Paired Observations; % Match = % of paired individuals who both agree about whether or not the classroom visited exhibited engagement, alignment, or rigor as defined by the EAR protocol.

© IRRE, 2010. 6

Source: Connell, JP., Klem, AM., Broom, JM, & Kenney, M. (2005). Going Small and Getting Smarter: Small Learning Communities as Platforms for Effective Professional Development. Prepared for the Office of Vocational and Adult Education, U.S. Department of Education, December, 2005.

Agreement among Expert Raters This study examined the extent to which expert raters agreed with each other on their ratings of

instructional practices occurring in high school classrooms. Pairs of 4 expert raters observed 19

different classrooms in one large urban high school in Houston, TX in Fall, 2006. Results indicated that

expert raters are highly consistent in their ratings of classroom practice: Engagement: 95% agreement;

Alignment: 74% agreement; Rigor: 89% agreement.

Agreement of Practitioners and Benchmark Raters An independent psychometric study of EAR conducted by a consortium of university researchers led

by Edward Deci at the University of Rochester and funded by IES established that both current district

instructional leaders in Arizona and independent former educators could be trained to criterion and

meet threshold levels of agreement with expert benchmark raters. Three types of classroom raters

were used for this study: benchmark, outside, and district. Benchmark raters are experienced

instructional specialists who formerly worked in public schools across the United States and who now

work for IRRE and use the EAR Protocol on a regular basis as a means of improving instruction in

schools. Outside raters are experienced educators who were trained by the benchmark raters to use

the EAR Protocol and have had no prior relation to IRRE or to the participating school district. District

raters are individuals who are employed by the schools district (typically principals, assistant

principals, and teacher-leaders). The district raters received extensive training by the benchmark

raters to use the tool to help improve instruction in their schools. Four high schools in a single district

participated.

After training was complete, the outside and district raters had three opportunities to make classroom

visits with benchmark raters from the IRRE staff: winter site visit (single day in January, 2009), spring

site visit (single day in March, 2009), and the March Benchmark week (4 days during which each study

class was visited two times). Inter-rater reliability results from those three time points appear in the

table below.

For both outside (independent) raters and current district educators there were high levels of

agreement with expert (benchmark) raters across each observation opportunity. The percentage of

rating decisions observers made across the three observation targets (engagement, alignment, rigor)

that agreed with expert rater decisions exceeded 80% and went as high as 90%.

© IRRE, 2010. 7

Table 2. Agreement on EAR Protocol Indicators between Expert Raters, Trained Independent Raters and Trained District Instructional Leaders

Outside (Independent) Raters District Raters

Winter Site Visit

Spring Site Visit

March Bench-mark

Winter Site Visit

Spring Site Visit

March Bench-mark

Number of Raters 2 2 3 28 26 12

TL Number Observations with a Benchmark Rater

10 10 86 105 123 57

Mean (SD) % Agreement1 .87 (.11) .90 (.09) .85 (.14) .86 (.12) .88 (.13) .83 (.12)

Kappa2 .72 .78 .69 .65 .75 .64

Validity Independent investigators (Edward Deci and Diane Early) are studying how changes in engagement,

alignment and rigor (at the classroom level) are associated with changes in student achievement as

measured by state assessments in ELA and mathematics in those same classes over a one year period.

This predictive validity study is part of a larger federally-funded national research project using

randomized trial design examining the efficacy of FTF’s instructional approach. The results of the EAR

predictive validity study will be available in the Winter 2010.

Face Validity

Over 18,000 classroom visits have been conducted using the EAR Classroom Visit Protocol over the

past 5 years and more than 800 instructional leaders in 17 districts and 131 schools have found this

assessment useful in guiding and bringing focus to their instructional improvement efforts. Schools

now implementing the latest version of EAR in Texas have shown the strongest academic gains after

one year of implementation of any cohorts in FTF history. The track record of successful

implementation and use by educators in diverse schools across the U.S. has shown that using EAR

frequently and with fidelity produces gains in achievement when the tool is used in concert with

ongoing professional development focused on engagement, alignment and rigor.

1 Proportion of 15 sub-indicators of engagement, alignment and rigor (EAR) on which two raters agreed that the sub-indicator was or was not present at threshold levels of quality. 2 Kappa was calculated for each observer, across all observations made with a benchmark. The value reported here is the average of the observer-level Kappas.

© IRRE, 2010. 8

DESCRIPTION OF MWM EFFECTIVE PRACTICES

MEASURING WHAT MATTERS© Institute for Research and Reform in Education

EFFECTIVE PRACTICE MWM DEFINITION SUPPORTING RESEARCH

Instructional Improvement (Engagement, Alignment, and Rigor)

Three overarching instructional goals are the focus of FTF instructional improvement:

Engagement – students actively process information in ways that indicate they are focused on the task at hand and interested in it.

Alignment – what is being taught and what students are being asked to do are aligned with the standards and curriculum, are on time and on target with the scope and sequence of the course of study and use methods of assessment that include those students will encounter in high-stakes testing.

Rigor – materials and instructional strategies challenge and encourage all students to produce work or respond at or above grade level. All students are required to demonstrate mastery at these levels and have the opportunity for re-teaching to mastery (Connell and Broom, 2004).

These three goals are the foundation of all professional development activities.

There is emerging consensus in the literature that for schools to be successful, they must provide students with supportive relationships, have high expectations for all students, and provide engaging and relevant instruction that is aligned with state standards (National Research Council and Institute of Medicine, 2004). Recent research has demonstrated that effective instructional practices that engage students in rigorous and meaningful academic content can be a mechanism to improve student outcomes (Gambone et al., 2004).

When instruction is more rigorous, students are more likely to report that teachers hold high expectations for them, that they know what it takes to succeed academically, and that teachers provided them with models of good work (Quint, Bloom, Black, and Stephens, 2005). In addition, when instruction is aligned with the needs of students they are more engaged in the material and motivated to learn (Charles A. Dana Center, University of Texas at Austin, 1999). At the same time, ensuring that instruction is aligned with state assessments helps push teachers to raise the rigor of instruction in a more focused and effective manner (Wagner, 2003).

Finally, research has also shown that student engagement in the classroom is one of the most robust predictors of student achievement and behavior in school (Willingham, Pollack, & Lewis, 2002; Lee & Smith, 1995; Lee, Smith, & Croninger, 1995; Marks, 2000; Ryan & Patrick, 2001; Steinberg, Brown, & Dornbusch, 1996).

© IRRE, 2010. 9


Effective Small Learning Communities (SLCs)

Thematic Small Learning Communities (SLCs) have 10-20 staff and no more than 325 students at the middle and high school levels. Students stay in their SLCs for most classes during the school day and across multiple years – e.g., all four years of high school or all three years of middle school. With a relatively small number of students, each SLC is a size that allows staff to know every student personally, but still large enough to include teachers who can ensure the needed breadth of content expertise. All high-school and many middle-school SLCs mix grades – with students from all grade levels in the building in each SLC – making it possible to base instruction and curriculum on individual academic needs and interests, rather than grade levels alone. SLC staff members study data on individual students in their SLC, take collective responsibility for every student’s success and make key decisions about discipline, staffing, time use, student scheduling and budget.

There is growing agreement that “smaller” is better when it comes to reforming large, urban high schools. Smaller learning environments such as those provided by small learning communities (SLCs) allow students and staff to develop strong relationships and create a setting more conducive to learning, especially for minority youth and those living in poverty (Klonsky, 1998; Gambone et al., 2004). Not only have SLCs been found to improve relationships between staff and students and among staff, but those stronger relationships, in turn, are linked to higher levels of engagement among both staff and students (Gambone et al., 2004; Quint et al., 2005). In contrast, large schools not only have a negative impact on the achievement of at-risk youth but have also been shown to widen the achievement gap between them and Caucasian students (Howley and Bickel, 2000).

In addition, common planning time, team teaching, and other forms of ongoing collaboration in SLCs allow teachers to improve their professional skills for instructing all students. Collective responsibility and cooperation among staff have been shown in national data to foster achievement gains between 8th and 10th grades (Moles, 2003; Lee & Smith, 1996).

© IRRE, 2010. 10


Use of Professional Development Small learning communities (SLCs) create opportunities for instructional staff – teachers, content area leaders, instructional coaches and supervisors – to energize and focus professional development by creating new venues and new incentives for teachers and other instructional leaders to learn about their own and others’ teaching practices. Faculty learn about new instructional strategies for strengthening student engagement, aligning instruction with state and local standards and incorporating high expectations for all students into their lesson planning and teaching practices. Typically, sessions include national instructional experts using simulations and demonstrations, faculty study and dialogue and opportunities for staff to create implementation plans and review ongoing support activities to put these new strategies into practice. Schools in later stages of implementation determine the focus for these capacity-building days by using assessments to pinpoint areas of need.

Professional development strategies that require teachers to work together and hold reflective conversations about their practice has been shown to improve teaching and learning (Knapp, McCaffrey, and Swanson, 2003). Such reflective conversations provide a forum for them to candidly and effectively discuss expectations, teaching strategies and student work. For instance, working toward better alignment of standards and curriculum depends teachers’ understanding of the connection between how they teach and how students perform on high stakes tests (Charles A. Dana Center, University of Texas at Austin, 1999).

In addition to improving instructional effectiveness, professional development strategies can enhance staff and student engagement. For instance, students in buildings with high-quality professional development were more likely to be engaged in school and to experience teachers as being more supportive and having high expectations for them (Gambone, et al., 2004).

FTF implementation was associated with a notable increase in the use of small-group teaching strategies in classrooms. In two independent evaluations of FTF schools, students in high schools were more likely to report having opportunities to work in teams on assignments and to work on interdisciplinary projects and projects connected to their futures and their lives outside school (Gambone, et al., 2004; Quint, et al., 2005).

© IRRE, 2010. 11


Family and Student Advocacy

First Things First SLCs include students’ families through the Family and Student Advocate System. This strategy puts a school adult in the corner of every student and strengthens the kind of family involvement that really matters for student achievement. Each staff member becomes an advocate for 15-20 students and their families, stays with them all the years they are in the school and does whatever it takes to help those students succeed. Family advocates contact families regularly and involve them, along with their child, in setting and meeting academic and behavioral goals.

There is no question that parent involvement in their child’s education helps that child do better in school (Eccles and Harold, 1996; Funkhouser and Gonzales, 1997; Henderson and Berla, 1994). However, not all involvement has an equal impact on achievement. The research findings indicate that parents can most effectively support their children in secondary school by creating a productive learning environment at home and advising or guiding their child’s academic decisions (Catsambis, 1998; Jordan, Orozco, & Averett, 2002; Plank & Jordan, 1997). Also important is parents’ attention to their child's status on important educational outcomes, both academic and behavioral, and their regular communication with adults who know their child at school. Essentially, when students, parents, and teachers/advocates work together, there is a greater likelihood that students will achieve success in school (Jordan, Orozco, & Averett, 2002).

All definitions are from the First Things First: From Getting Off the Dime to Getting It on the Ground report for the Department of Education. Instructional Improvement references are from (Gambone, M.A., Klem, A.M., Summers, J.A., Akey, T. and Sipe, C., 2004).

© IRRE, 2010. 12

MWM PROTOCOLS, SURVEYS AND SELF-ASSESSMENTS


Institute for Research and Reform in Education

Instructional Improvement

Description of Protocol Types of Reports Available

Engagement, Alignment and Rigor (EAR) Classroom Visit Protocol

PDA-based rating form used during 20-30 minute classroom visits by individuals or groups

Classes and teachers to be visited pre-loaded for each visitor

Customized fields available for district-specific learning materials and activities

Protocol includes categorical (Y/N) and percentage ratings for three instructional goals: engagement, alignment and rigor (with sub-indicators)

Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation

Visitor prompts on PDA available using materials from trainings to refresh rater

Stylus navigation with scrolling and up and down buttons

Lead rater capability during calibration visits using infra red links among visitors’ PDAs

Time-, date-, and duration- coded individual visit reports:

Learning materials and activities used

Engagement, alignment and rigor of teaching and learning

Sub-indicator ratings for each instructional goal.

Global quality ratings across instructional goals Aggregated reports by:

Time period

Classroom visitor

Teacher visited

Small learning community

Content area

Building

District Currently available on PDA and scanable form

© IRRE, 2010. 13



Peer Observation and Dialogue: Quality Assessment

PDA-based rating form used during 45- minute teacher meeting following peer classroom visits held during common planning time, late start/early release or professional development days SLCs and departments and number of teachers in each to be visited pre-loaded for each visitor Protocol includes categorical (Y/N) and tallied ratings of the meeting’s quality and productivity Visitor prompts available using materials from trainings to refresh raters Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation Stylus navigation with scrolling and up and down buttons

Quantitative reports with time, date, meeting type and duration. Reports include:

Degree of follow-up and follow-through by participants from previous meetings.

Levels and quality of participation by staff

Quality of teacher reports on their observations

Specificity, alignment and accountability mechanisms of action plans based on peer observation and dialogue

Aggregated reports of the above by time periods, visitor, SLC, department, building, district

Currently available in scanable form. PDA version in development

Use of the Tuning Protocol to Study Student Work and Improve Practice: Quality Assessment

PDA-based rating form used during 45-90 minute teacher meeting following review of student work using the tuning protocol held during common planning time, late start/early release or professional development days SLCs and departments and number of teachers in each meeting to be visited pre-loaded for each visitor Protocol includes categorical (Y/N) and tallied ratings of the meeting’s quality and productivity Visitor prompts available using materials from trainings to refresh raters Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation Stylus navigation with scrolling and up and down buttons




Quality of teacher reports on their use of the tuning protocol

Quality of teacher preparation and materials for meeting

Specificity, alignment and accountability mechanisms of action plans based on use of the tuning protocol

Aggregated reports of the above by time periods, visitor, SLC, department, building, district Currently available in scanable form. PDA version in development

© IRRE, 2010. 14



Alignment and Rigor Through Common Assessments and Evaluations of Student Work: Quality Assessment

PDA-based rating form used during 45- minute teacher meeting focused on increasing alignment and rigor by building common assessments and evaluations of student work held during common planning time, late start/early release or professional development days SLCs and departments and number of teachers in each to be visited pre-loaded for each visitor Protocol includes categorical (Y/N) and tallied ratings of the meeting’s quality and productivity Visitor prompts on PDA available using materials from trainings to refresh raters Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation Stylus navigation with scrolling and up and down buttons




Progress of group to generate common assessments and evaluations of student work

Quality ratings of common assessments (EAR) and evaluations of student work (AR) produced

Specificity, alignment and accountability mechanisms of action plans based on use of common assessments and evaluations of student work

Aggregated reports of the above by time periods, visitor, course of study, department, building and district Currently available in scanable form. PDA version in development.

© IRRE, 2010. 15

FAS Description of Protocol Types of Reports Available

Family Advocate Period Visit Protocol

PDA-based rating form used during 20 minute family advocate period visits to assess:

Active engagement of students in family advocate period activities,

Alignment of activities with FAS goals

Quality of relationships exhibited (including mutual accountability and follow-through)

Rigor of advocacy period activities Advocate names to be preloaded for visitors Protocol includes frequency and duration of family advocate period Customizable fields available for capturing district-specific FAP activities used Protocol includes categorical (Y/N) and percentage ratings; Stylus navigation Visitor prompts available to refresh raters on training material Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation

Quantitative reports on specific visits that are time, data, and duration coded including:

Learning activities and materials used

Source of learning activities (teacher, FAS guide, other)

Student engagement, alignment with FAS goals, quality of relationships, and rigor

Aggregated reports by:

Advocate

SLC

Building

District Currently available on PDA and scanable form.

Family Advocate System Discussions During Common Planning Time

PDA-based rating form used during 45- minute teacher meeting in which family advocate reports and discussion of student progress are presented. Teacher meetings held during common planning time, late start/early release or professional development days. SLCs and departmental meetings to be visited pre-loaded for visitors Protocol includes categorical (Y/N) and tallied ratings of the meeting’s quality and productivity Visitor prompts available using materials from trainings to refresh raters Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation




Focus and balance of advocate reports

Coverage of students/families identified for discussion

Specificity and accountability mechanisms of action steps based on discussion of student/families

Aggregated reports of the above by time periods, visitor, SLC, department, building, district Currently available in scanable form.

© IRRE, 2010. 16

SLC Effectiveness


SLC Target-Setting and Action Planning Around Student Outcomes: Quality Assessment

PDA-based rating form used during 45- minute teacher meeting focused on SLCs setting student outcome targets and using student outcome results for action planning during common planning time, late start/early release or professional development days SLCs and departments and number of teachers in each to be visited pre-loaded for each visitor Protocol includes categorical (Y/N) and tallied ratings of the meeting’s quality and productivity Visitor prompts available using materials from trainings to refresh raters Data synced with PC using web-based applications and sent to IRRE server for analysis and report generation Stylus navigation with scrolling and up and down buttons




For target setting and action planning sessions: Quality of process and products

Aggregated reports of the above by time periods, visitor, SLC, building and district

Currently available in scanable form. PDA development is underway.

© IRRE, 2010. 17

SLC Functioning Description of Surveys Types of Reports Available

Staff Surveys

Self-report surveys for teachers and building leaders. The survey captures information on the extent to which teachers and building leaders receive the pressure, support, and flexibility necessary to implement well functioning SLCs and improve instruction. Surveys are provided in scanable paper form or can be accessed through a web-based application.

Quantitative reports present information on:

Extent and type of supports provided by peers and by system leaders (e.g., central office, building administration) supporting implementation of effective practices.

Extent to which expectations for staff behavior – and student outcomes associated with those behaviors – are made clear by system leaders.

Extent to which consequences for not meeting district expectations are made clear by system leaders.

Time available for professional development (e.g., common planning time, early release/late start, in –service days); whether that time is used effectively; whether training provided was effective.

Level of commitment to reform from school staff and system leaders.


SLC

Content Area

Building

District Currently available via secure web-based survey or in scanable form.

© IRRE, 2010. 18

FAS Description of Surveys Types of Reports Available

Student, Parent, and Advocate Surveys

Brief 5-10 minute self-report surveys for students, parents, and advocates assessing the extent to which schools are fully implementing and achieving goals of Family Advocate System (FAS). Student survey captures information on quality of relationship with family advocate; family advocate period, one-on-one meetings with family advocate, and family conferences. Parent survey captures information on:

Frequency of contact

Quality of relationship with family advocate and of family conferences

Effectiveness of FAS Advocate survey captures information on:

Frequency and quality of family contacts, family advocate period, family conferences, and one-on-one meetings with students.

Effectiveness of FAS and of the training to implement FAS. Surveys are provided in scanable paper form or can be accessed through a web-based application.

Quantitative reports as reported by student, parents, and advocates including:

Quality of family advocate period, family conferences, and one-on-one meetings between advocate and student;

Frequency of contacts

Quality of relationships between advocate and student and advocate and parents

Extent to which FAS has made a difference Aggregated reports by:

Advocate group

SLC

Grade Level

Building


© IRRE, 2010. 19


Description of Surveys

Types of Reports Available

Student Survey of Instructional Experience and Teacher Support

Brief 10-15 minute self-report surveys for students assessing the extent to which students perceive an engaging and rigorous instructional experience and support from their teachers. Sample instruction-related items include:

Students understand why the work they are doing is important.

Students talk about connections between our work in school and what is going on in our lives outside of school.

The work I do in my _______ class is usually interesting to me.

The teacher of my _______ class gives students opportunities to learn more about a topic we are interested in.

My teachers make clear the connections between what we are learning and what we will need to do on the state assessment.

Rubrics are posted in my _____ class so I know what is expected of me.

My teachers make clear to me what I am expected to know and be able to do.

My teacher makes sure students understand what we have just learned before moving on to a new topic.

Sample teacher support items include:

My teachers interrupt me when I have something to say.

My teachers care about how I do in school.

My teachers don’t seem to have enough time for me.

Surveys are provided in scanable paper form or can be accessed through a web-based application.

Quantitative reports as reported by students including:

Percent of students reporting experiencing threshold levels of engagement, alignment and rigor during instruction

Percent of students reporting experiencing threshold levels of engagement OR alignment OR rigor

Percent of students reporting no experience of engagement, alignment or rigor

Percent of students perceiving threshold levels of support from teachers


Advocate group

SLC

Grade Level

Course of study

Building


© IRRE, 2010. 20

EAR CLASSROOM VISIT PROTOCOL© at a GLANCE



Features Types of Reports Available

PDA-based rating form used during 20-30 minute classroom visits by individuals or groups

All classes and teachers schedules are pre-loaded on PDA for each visitor

Customized fields available for district-specific learning materials and activities

Protocol includes categorical (Y/N) and percentage ratings for three instructional goals: engagement, alignment and rigor (with sub-indicators)

Visitor prompts on PDA available using materials from trainings to refresh rater

Stylus navigation with scrolling and up and down buttons

Lead rater capability during calibration visits using infrared link among visitors’ PDAs

Data collected is uploaded to M\WM server using PC internet connection for analysis and report generation

Users access MWM secure website for access to reports

Time-, date-, and duration- coded individual visit reports:

Learning materials and activities used

Engagement, alignment and rigor of teaching and learning

Sub-indicator ratings for each instructional goal.

On-line aggregate reports by:

Time interval (e.g. week, month, quarter, year)

Classroom visitor

Teacher visited

Learning community

Content area

Building

District

© IRRE, 2010. 21

ENGAGEMENT, ALIGNMENT AND RIGOR (EAR)© CLASSROOM VISIT PROTOCOL

School Name: __________________________ Name of SLC: ________________________________________

____________________’s ____________________________ class was visited on ______________________ (teacher) (subject) (date)

at ________ for ______ minutes. # of students in the class: ______ # of visitors: ______ (time) (#)

Names of all team visitors____________________________________________________________________ (Please list all visitors and then circle your name)

DESCRIPTION OF VISIT FINAL VISITOR PROMPT 1. Instructional goals assessed

_____ engagement

_____ alignment

_____ rigor

_______ _______ _______

Engagement: Students are actively processing information (listening, reading, thinking, making) or communicating information (speaking, performing, writing) in ways that indicate they are on task and interested. Alignment: What is being taught and what students are being asked to do are aligned with the standards and curriculum; are “on time” and on target with the scope and sequence of the course of study; and provide students opportunities to experience high stakes assessment methodologies among other assessment approaches. Rigor: Learning materials and instructional strategies being used challenge and encourage all students to produce work or respond at or above grade level. All students are required to demonstrate mastery at these levels and have the opportunity for re-teaching to mastery.

2. Learning materials used (select all observed):

Textbook ----- Writing (journal)

Worksheet ----- Writing (respond to prompt)

Text read aloud ----- Classroom library

Text shared Reading ___ Academic Journals

Video clips/picture cards ----- Key vocabulary emphasized

Mini-lecture ----- Overheads

Read 180 (books on tape + leveled text + computer)

Computer ----- Internet

Leveled text ----- Instruction software

Problem of the day ___ Chart Paper

White board ----- Calculators/Graphing Calculators

Math/Literacy tool kit ----- Other_____________________ Bilingual dictionaries/texts when appropriate-ESL

----- Hands on materials/manipulatives

What materials is the teacher using to build understanding around the content?

Check off ALL learning materials observed. Circle PREDOMINANT learning materials observed.

3. Learning activities used (select all observed):

----- Individual work sheets ----- Individual writing

----- Think-pair-share ----- Reading rotations

----- Silent reading ----- Individual projects

----- Lecture with note-taking ----- Small group guided instruction

----- Whole class disc/Q&A ----- Whole group guided instruction

----- Group work ----- Group projects

----- Writer‟s workshop ----- Comprehensible input

----- Math rotations ----- Engagement strategies

----- Uses modeling, visuals, gestures to clarify concepts

___ Student Demonstrations ___ Other ____________________

What learning structures and activities are evident?

Check off ALL learning activities and structures observed.

Circle PREDOMINANT learning activities or structures observed.

© IRRE, 2010. 22

ENGAGEMENT FINAL VISITOR PROMPT

1. _____ % of students are on task. ______ Scan the room periodically and estimate the percentage of students who appear to be on task: thinking, speaking, writing, making, or listening.

2. _____ % of students are actively engaged in the work requested.

_______ Look more closely at a RANDOM SAMPLE of students who are on task: what percentage of these students appear interested in the work?

3. Of the _____ (#) students visited individually:

a. _______ (#) are on task and

b. _______ (#) are actively engaged in the work requested.

______

______

______

Choose three to five students randomly and ask them if it‟d be alright if you talked to them about what they‟re doing.

Interview probes:

1) What are you working on?

2) What does teacher expect you to learn by doing this work?

3) Why do you think this work is important?

4) How interesting is this work to you? ? Not at all? A little? Really interesting?

Code student as on task if probes 1 and 2 match with observed assignment and expectations.

Code student as actively engaged if probe 3 is answered cogently and 4 is answered at least a “little interesting.”

ALIGNMENT FINAL VISITOR PROMPT

1. The learning materials:3

a. ___ do ___ do not reflect content standards guiding this class.

b. ___ are ___ are not aligned with the designated curriculum to teach those standards.

c. ___ are ___ are not aligned with the pacing guide of this course or grade level curriculum.

______ ______ ______

Observe the learning materials (those there at the beginning of the visit and introduced over the course of the visit) – review learning materials you checked earlier.

Are those materials aligned:

1) with the district content standards covered by this course?

2) with the designated district curriculum for this instructional unit?

Using the district syllabus or pacing guide for this course of study, determine whether the instructional materials are “on time.”

2. The learning activities:1

a. ___ do ___ do not reflect content standards guiding this class.

b. ___ are ___ are not aligned with the designated curriculum to teach those standards.

c. ___ are ___ are not aligned with the scope and sequence of the course according to the course syllabus.

______ ______ ______

Observe the learning activities underway and initiated during the visit – review learning activities you checked earlier.

Do these activities incorporate content aligned with the district standards to be covered in this course of study? Are they aligned with the designated district curriculum to teach these standards?

Using the syllabus or pacing guide for this course of study, determine whether the learning activities are “on time.”

3. The student work expected ___is ___ is not aligned with the types of work products expected in state grade level performance standards.

______ Observe the kinds of student work expected (questions to answer, written work, oral reports, demonstrations).

Are students being asked to complete work that aligns with the kinds of work products expected to meet or exceed state and district grade level standards?

3 Teacher interview may be required to complete this assessment.

© IRRE, 2010. 23

ALIGNMENT FINAL VISITOR PROMPT

4. Student work ___ does ___ does not provide exposure to and practice on high stakes assessment methodologies.

______ Observe the summative and ongoing assessments completed by students during the visit.

Are students getting opportunities to practice assessment methodologies used in state tests and other high stakes assessments?

RIGOR FINAL VISITOR PROMPT

1. The learning materials ___ do ___ do not present content at an appropriate difficulty level.

______

Are learning materials observed (review learning materials checked) providing challenging enough content to support all students completing grade level or better work according to state and district performance standards?1

2. The student work expected ___ does ___ does not allow students to demonstrate proficient or higher levels of learning according to state grade level performance standards.

______ Observe student work being displayed and work requested during and following learning activities (review learning activities checked):

What level of thinking and performing do the learning activities require and does the student work observed reflect:

Basic? Intermediate? Advanced?

Code student work expected as rigorous only if preponderance of observed work and work expected during classroom visit were at Intermediate Level and some of the work was at the Advanced Level.

3. Evaluations/grading of student work ___ do ___ do not reflect state grade level performance standards.4,5

______ Ask the teacher how they evaluate student work that “counts” toward the students‟ course grade. 1. Are grading procedures standards based and appropriately tied

to progress toward and achievement of proficient work according to state grade level standards and objectives for the course?

2. Are grading rubrics and examples of student work displayed and/or easily accessible to students?

3. Do rubrics and examples demonstrate proficient and exemplary work according to state grade level standards and objectives for the course?

Choose two or three students randomly and ask them: 4. whether and

how they use rubrics to judge the quality of their and others‟ work; 5. to show you a piece of their work on the topic they‟re working on now that they are proud of and explain what their grade was and why they got it. Code evaluations of student work as rigorous if probes 1-5 are either „yes‟

(probes 1-3) or cogently responded to by all students (probes 4-5).

4. ____% of students were required to demonstrate whether

or not they had mastered content being taught.

______ Observe whether and how teachers extract information on student mastery of content (oral response to questions, written answers, demonstrations)

Look at how the teacher ensures that these students had made adequate progress toward mastery using this „assessment‟ technique?1

Cooperative learning/evaluation structure?

Teacher or peer conferences?

What percentage of students were involved and inspected in these demonstrations of mastery?

5. ____% of students demonstrated threshold levels of

mastery before new content was introduced.

______ Note: If no new content introduced – check N/A. Depending on ongoing or summative assessment technique observed, take representative sample of results (oral answers to teachers‟ questions, written responses to problem, student conference discussions) to identify percentage of students who demonstrated expected levels of mastery before new content was introduced.

4 Teacher interview may be required to complete this assessment.

5 Observer may need to examine the tests or assessments used after the observation is completed.

© IRRE, 2010. 24

SCREEN SHOTS OF EAR ON POCKET-PC DEVICE

SAMPLE EFFECTIVE PRACTICES REPORTS: EAR

IRRE, 2007. 26

EAR Classroom Visit Report – Hope High School: Small Learning Community (by Teacher Classrooms)

School


# Teacher Classrooms

% of Teacher Classrooms

Meeting Threshold

Meets Threshold for

HQTL*



Meeting Threshold

Meets Threshold for

HQTL*



Meeting Threshold

Meets Threshold for

HQTL*

Hope High School 70 56% 70 45% 70 29%

Small Learning Communities (SLCs)



Meeting Threshold

Meets Threshold for

HQTL*



Meeting Threshold

Meets Threshold for

HQTL*



Meeting Threshold

Meets Threshold for

HQTL*

Business 17 47% 17 12% 17 6%

Engineering & Trade 16 81% 16 81% 16 44%

Fine Arts 19 47% 19 53% 19 37%

Media & Technology 18 61% 18 33% 18 28%

* At least 50% of the Teacher Classrooms need to meet the [EAR] threshold in order for the district, school or small learning community to meet the threshold for High Quality Teaching & Learning.

IRRE, 2007. 27

EAR Classroom Visit Report – Hope High School: Small Learning Community(by Teacher Classrooms)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Perc

en

t o

f T

each

er

Cla

ssro

om

s M

eeti

ng

th

e T

hre

sho

ld

Business Engineering & Trade Fine Arts Media & Technology

IRRE, 2007. 28

EAR Classroom Visit Report – Hope High School: Small Learning Community (by Visits)

School


# Visits % of Visits

Meeting Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL*

Hope High School 378 64% 378 56% 378 11%

Small Learning Communities (SLCs)


Meeting Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL*

Business 92 42% 92 32% 92 11%

Engineering & Trade 104 81% 104 84% 104 50%

Fine Arts 89 60% 89 62% 89 44%

Media & Technology 93 74% 93 48% 93 28%

* At least 75% of the Visits need to meet the [EAR] threshold in order for the district, school or small learning community to meet the threshold for High Quality Teaching & Learning.

IRRE, 2007. 29

EAR Classroom Visit Subindicator Report – Hope High School

District School SLC Department Course

Hope ISD Hope High School All All All

Engagement % of Visits Meeting

Threshold6

1. ___ % of students were on task. 71%

2. ___ % of students were actively engaged in the work requested. 62%

3. ___ (#) students were visited individually. 945

3a. Of the ___ (#) students visited individually ___ (#) were on task. 677

3b. Of the ___ (#) students visited individually ___ (#) were engaged in this work. 425

Alignment % of Visits Meeting

Threshold

1a. The learning materials ___ did ___ did not reflect content standards guiding this class. 80%

1b. The learning materials ___ were ___ were not aligned with the designated curriculum to teach those standards 82%

1c. The learning materials ___ were ___ were not aligned with the pacing guide of this course or grade level curriculum 75%

2a. The learning activities ___ did ___ did not reflect content standards guiding this class. 78%

2b. The learning activities ___ were ___ were not aligned with the designated curriculum to teach those standards 75%

2c. The learning activities ___ were ___ were not aligned with the scope and sequence of the course according to the course syllabus.

77%

3. The student work expected ___ was ___ was not aligned with the types of work products expected in state grade level performance standards

60%

4. Student work ___ did ___ did not provide exposure to and practice on high stakes assessment methodologies. 52%

Rigor % of Visits Meeting

Threshold

1. The learning materials ___ did ___ did not present content at the appropriate difficulty level. 60%

2. The student work expected ___ did ___ did not allow students to demonstrate proficient or higher levels of learning according to state grade level performance standards.

48%

3. Evaluations/grading of student work ___ did ___ did not reflect state grade level performance standards. 54%

4. ___ % of students were required to demonstrate whether or not they had mastered content being taught. 35%

5. ___ % of student demonstrated threshold levels of mastery before new content was introduced. 33%

6 378 Total Visits

IRRE, 2007. 30

7 92 Total Visits



Hope ISD Hope High School Business All All


Threshold7







Threshold






2c. The learning activities ___ were ___ were not aligned with the scope and sequence of the course according to the

course syllabus. 74%

3. The student work expected ___ was ___ was not aligned with the types of work products expected in state grade level

performance standards 60%



Threshold


2. The student work expected ___ did ___ did not allow students to demonstrate proficient or higher levels of learning

according to state grade level performance standards. 44%




IRRE, 2007. 31

EAR Classroom Visit Report – Hope High School: Department (by Visits)

School



Meeting Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL*

Hope High School 378 60% 378 56% 378 36%

Department / Content # Visits % of Visits

Meeting Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL*

Language Arts 95 59% 95 57% 95 39%

Mathematics 95 79% 95 80% 95 52%

Science 75 63% 75 52% 75 45%

Social Studies 75 41% 75 35% 75 9%


IRRE, 2007. 32

EAR Classroom Visit Report – Hope High School: Department(by Visits)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Perc

en

t o

f V

isit

s M

eeti

ng

Th

resh

old

Language Arts Mathematics Science Social Studies

IRRE, 2007. 33

8 75 Total Visits



Hope ISD Hope High School All Social Studies All


Threshold8







Threshold








3. The student work expected ___ was ___ was not aligned with the types of work products expected in state grade

level performance standards 40%



Threshold







IRRE, 2007. 34

EAR Classroom Visit Report – Hope High School: Department: Course (by Visits)

School: Department



Meeting Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL*

Hope High School 378 60% 378 56% 378 36%

Social Studies 75 41% 75 35% 75 9%

Course # Visits % of Visits

Meeting Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL* # Visits

% of Visits Meeting

Threshold

Meets Threshold for

HQTL*

Economics 15 20% 15 40% 15 13%

US Government 15 20% 15 20% 15 7%

US History 20 20% 20 10% 20 0%

World History 20 40% 20 50% 20 20%


IRRE, 2007. 35



Hope ISD Hope High School All Social Studies US History


Threshold







Threshold








3. The student work expected ___ was ___ was not aligned with the types of work products expected in state grade

level performance standards 40%



Threshold







IRRE, 2007. 36

Percent of Visits Meeting EAR Thresholds for Your District

District School SLC Teacher Subject Area Visitor Begin

Date

End

Date

Hope ISD ALL ALL ALL ALL ALL 1/1/2005 6/1/2005


Threshold




Threshold






2c. The learning activities ___ were ___ were not aligned with the scope and sequence of the course according to the course

syllabus. 40%

3. The student work expected ___ was ___ was not aligned with the types of work products expected in state grade level

performance standards 45%



Threshold


2. The student work expected ___ did ___ did not allow students to demonstrate proficient or higher levels of learning according to

state grade level performance standards. 73%




IRRE, 2007. 37

TRAINING FOR MWM EFFECTIVE PRACTICES: EAR CLASSROOM VISIT PROTOCOL



ORIENTATION An overview of the systematic and systemic approach to measuring the effectiveness of the strategies used to improve student outcomes. This training includes:

An introduction and overview of the purpose, the approach and the materials used to measure student outcomes and instructional practices.

Previewing sample reports that can be generated from the measuring what matters material.

Developing a common understanding of the definitions for engagement, alignment and rigor

Examining the classroom visit protocol; the indicators, the visitor prompts, and how it is used.

Looking at classroom scenarios and beginning the conversations around engagement, alignment and rigor.

Visiting classrooms to begin calibrating our understanding of the definitions and the work in practice

Developing group data from the engagement, alignment and rigor conversations to determine where further supports needed.

Looking at system leadership and each individuals role in the implementation of the system.

Collectively developing a uniform communication plan to take back to the district and schools.

CALIBRATION/PDA This training provides the opportunity to go more in-depth into the definitions of engagement, alignment and rigor, the protocol and the specific content indicators. Teams of participants conduct classroom visits using the protocol and participate in conferences to discuss and calibrate their thinking around each of the indicators. The PDA training actually begins with the training of technology volunteers at each schools, learning how to install the PDAs and the software that supports the use of Measuring What Matters. The afternoon of this full day training is on the use of the PDA for gathering data on the state of teaching and learning in the school district. This training includes:

IRRE, 2007. 38

A review of the definitions and the protocol allowing for further questions and discussion around each of the indicators.

A presentation of specific content indicators to look for while doing the classroom visits.

A large group video simulation to create a common understanding of the process

Teams of participants conducting classroom visits lead by a trained facilitator

Calibration conferences to discuss and come to common understandings what is being looked for and how is it determined.

Large group discussions around the calibration process

Preparation for calibrations visits and data collection. The data from the ten calibrations visits will provide feedback on the needs for further calibration and discussions.

Familiarizing the participants with the PDA and how it works

Developing the skills necessary to use the PDA to do both individual and calibration classroom visits.

TEACHER ORIENTATION – This is a ninety minute overview of the Measuring What Matters process, the indicators that observers are looking for and what effect this process has for them. More in-depth training and discussions on engagement, alignment and rigor will be conducted in SLC meetings. This training includes:

An introduction and overview of the purpose, the approach and the materials used to measure student outcomes and instructional practices.

Previewing sample reports that can be generated from the measuring what matters material.

Developing a common understanding of the definitions for engagement, alignment and rigor.

Looking at the indicators and what visitors will be measuring.

Discussing how this will effect their classroom practice. DATA CALIBTRATION/REFLECTIVE CONVERSATIONS – At this point in the process, data has been generated from the team’s ten classroom visits. This data provides a clear picture of teams’ common understanding of engagement, alignment and rigor and where further discussion and understanding needs to take place. The visits also provide the platform to think about the types of conversations that can take place with teachers to begin reflecting about their practice. Entering those conversations and framing the reflective questions with the purpose of inviting teachers into the challenges and solutions is the goal.

IRRE, 2007. 39

This training includes:

Clarification on specific indicators and the visitor prompts and the process for gathering information.

Examining the data generated from the calibration visits and discussion around the definitions and expectations.

Participate in data-driven dialogue using the calibration data.

Learn to enter into conversations and frame questions that move teachers from being dependent to interdependent in their practice

Learn the elements of a reflective question

Determine the priority of the issues to be addressed and how to address them

Practice framing effective questions

Plan for delivery and follow-up of the questions and conversations

A charge to the participants to complete a specific number of visits prior to the evaluation and fine tuning training.

ACCESSING AND USING THE DATA – This training takes place in a setting in which participants have access to computers and the data. It is in part a technical training in which participants learn to access and generate reports from the classroom visits for specific purposes as well as reports on student outcomes. It then moves to how to use this data to influence classroom practice. This training includes:

A review of the reports available.

A step by step simulation on finding an generating reports

A guided practice in determine what information in needed, how to find that information and then, how to put it into reports that will beneficial.

An opportunity to generate reports from the data collected during the calibration visits on each campus.

Small group discussions determining what reports are needed for specific purposes and then generating those reports.

Looking at the data collected from the visits and conducting data-driven discussions and how this data will be used to develop action plans.

Determining when to introduce the reports to the SLC’s and guiding the SLC’s through the process of data-driven discussions and action planning.

ON-GOING COACHING AND MONITORING – IRRE will make frequent district and school visits to work with the administration and teachers in making this process as effective as possible in increasing student achievement.

IRRE, 2007. 40

TECHNOLOGY SUPPORTS AND CONSULTATION FOR EFFECTIVE PRACTICES



ASSESSMENT OF TECHNOLOGY INFRASTRUCTURE AND STATE OF DATA—Appropriate district leaders respond to a series of questions posed by the MWM technology team in order to complete a thorough assessment of the structure and capacity of the District’s technology to support the development of student outcome reports and to ensure that PocketPC technology (Effective Practices on PDA) will operate effectively. Following submission of the interview form, IRRE staff schedule and conduct a consultation to review the interview responses. This consultation includes:

District administrators complete the MWM Technology Infrastructure Interview with IRRE staff

IRRE staff provide specifications on recommended PDA devices to be used for Effective Practices data collection

Identify the District administrators accountable for technology and building-level master schedules

Establish general timeline for PDA selection, ordering, delivery and set-up

Discuss possible deployment models including IRRE staff set-up or site-based set-up using school-based Technology Liaisons in each building

Obtain contact information for principals PLANNING AND LOGISTICAL SUPPORT FOR DEPLOYMENT—MWM staff provides three options for logistical support to deploy PDA devices to trained users: 1) On-site MWM staff support; 2) On-site Technology Liaison training and off-site follow-up; 3) MWM staff provide off-site support. The determination of which option is best suited to the site’s needs is based on: internal capacity to manage technology installations and deployment; the time available to local staff; and experience installing and supporting PDA devices.

1. On-site MWM staff support. This consultation includes:

Planning calls with district-designated technology point person to begin process of identifying PDA devices to purchase; discussing PDA set-up process; identifying timeframe for deployment and Technology Liaison training; and order-shipment timeframe required

IRRE, 2007. 41

MWM staff assistance to the district with creation of PDA tagging process and inventory management

Consultation with building principals to identify school-level technology liaison, outline technology liaison roles and responsibilities for deployment and post-training support; describe technical/security requirements for deployment; and request location for technology liaison training

MWM preparation and shipment of Effective Practices Resource Manual to be distributed following PDA training

Consultation and on-site training with principal-designated technology liaisons to prepare them to manage their building’s installations of PDAs

MWM staff collaborate with District IT staff to install central office user PDAs

MWM staff conduct site-visits to each building to inspect and trouble-shoot installations of PDAs

2. On-site Technology Liaison Training and Off-Site MWM Support. This

consultation includes:




MWM staff preparation and shipment of Effective Practices Resource Manual to be distributed following training for users

On-site consultation and training with technology liaisons to assist them with their building’s installations of PDAs

Phone and email consultation with District IT staff to install central office user PDAs

MWM staff are available via phone or email to assist Technology Liaisons both during and following installation of PDAs

3. Off-Site MWM Support. This consultation includes:


IRRE, 2007. 42



MWM staff preparation and shipment of Effective Practices Resource Manual to be distributed following PDA training for users

Phone and email consultation with technology liaisons to assist them to conduct installations of PDAs in their school

Phone and email consultation with District IT staff to install central office user PDAs

MWM staff are available via phone or email to assist Technology Liaisons both during and following installation of PDAs

Each planning and logistics support option includes:

Access to 1-800-560-IRRE Help Line (ext 1 for MWM Technical Support) for live support with guaranteed 4 hour response

Effective Practices Resource Manual for each trained user. The manual includes all information provided during training as well as documentation, frequently asked questions, and tips for using the PDA device and the Effective Practices software.

PREPARING DATABASES FOR CUSTOMIZED PDA USE—MWM staff work with district and/or school staff to obtain lists of trained users and the school’s master schedule for the fall and spring semesters. This consultation includes:

Obtaining lists of trained users for each building

Assigning log-on user id’s and passwords and access privileges for each user

Obtaining master schedules for each school, verifying the information for missing or incorrect course information, working with district or school staff to resolve problems

Uploading the user lists and master schedules into IRRE’s database

Confirming successful upload and readiness for PDA use

Sending confirmation and readiness to users that data collection may begin

ONGOING SUPPORT AND TECHNICAL ASSISTANCE – MWM staff and our technology vendor are available to troubleshoot and resolve problems with either the PDA device

IRRE, 2007. 43

or the Effective Practices tools. Among the technical support activities occurring on an on-going basis are:

Quality control samplings of data being collected via the PDA to ensure that the database is capturing all data uploaded to the IRRE server

Providing daily staffing of the MWM technical support hotline so users can call or email with questions

Periodic conference calls with local site managers to collect an independent accounting of the number of observation visits being uploaded to the IRRE server and to stay on top of emerging problems with the Effective Practices application and observation protocols;

Availability to improvements upgrades or fixes to the Effective Practices application by accessing the mwm.irre.org website. All users receive email notifications when upgrades or fixes become available and detailed instructions for how to update or upgrade their existing protocols

Phone calls to all MWM technology liaisons and FTF site coordinators when upgrades, updates and fixes are available and instructions for how to assist users in completing an upgrade to the existing tools

In the event that a PDA becomes disabled, lost or stolen, users can continue to collect data using paper/pencil versions of the tool and have two options for uploading the data to the IRRE database:

o Entry directly into the IRRE database using the Enter a Paper

Protocol tool at the mwm.ire.org website o Submitting the paper version, which is in a scanable format, to

MWM staff for scanning and upload into the IRRE database

Continued refinements to each Effective Practices application to ensure that users are able to use highly relevant, useful and reliable tools

IRRE, 2007. 44

FREQUENTLY ASKED QUESTIONS ABOUT THE EAR CLASSROOM VISIT

PROTOCOL©

1. Is the engagement, alignment and rigor (EAR) data used as part of teachers’ personnel evaluation? No, this process has been developed to identify areas of strengths, needs, and growth within schools, small learning communities/academies, and the district based on the First Things First framework guiding our reform effort. However, the information obtained may be used by the Principal or district level administrator to make changes in the supports provided as part of this initiative. 2. What qualifies visitors to make judgments about the quality of teachers’ instruction?

Every person who participates in this process has been trained in the procedures and the tools necessary to successfully look at areas of strengths, needs, and growth based on the First Things First framework. 3. How can this help teachers improve the quality of their instruction? Measuring What Matters visits can benefit the quality of instruction in several ways by:

offering feedback from coaches and other instructional leaders after classroom visits.

providing an opportunity to discuss with a trained visitor challenges and questions that teachers find in their classroom.

providing a clear picture of a small learning community’s strengths and challenges so that professional development is focused in a way that is useful and relevant for all teachers.

By having a common definition of what good teaching and learning looks like (EAR) and having reliable information being gathered on the quality of teaching and learning in classrooms (the classroom visit protocol and reports), teachers and coaches can more effectively develop strategies for improving engagement, alignment and rigor in their classrooms and tracking the effectiveness of these strategies. 4. Where will the information from the EAR protocol be sent once it is collected? The information recorded on either the paper and pencil or PDA versions of the classroom visit protocol will be sent electronically to MWM’s secure server outside the District. The information on this server will be accessible by MWM staff, by the classroom visitor who sent it and by instructional leaders.

IRRE, 2007. 45

5. How often can teachers expect visitors in their classrooms? Classroom visits will occur on a regular basis as part of the district’s ongoing assessment of its progress in improving teaching and learning. Visit frequency will be based on instructional leaders’ and individual teachers’ judgments about their usefulness in strengthening specific areas of instruction. 6. Will teachers get to see the results of visits to their classrooms? Yes. The purpose of the visits is to help teachers refine their instructional skills and to suggest ways to improve the quality of teaching and learning. A second purpose is to guide and assess the effectiveness of the supports provided by coaches and supervisors to teachers. In order for this to happen, teachers, coaches and supervisors need feedback about what was seen in classrooms. 7. How will teachers get information back about visits to their classrooms?

Shortly after each visit, at least one of the classroom visitors will meet with the teacher to discuss the information they gathered during their time in the classroom. They’ll talk specifically about which areas of instruction were the focus of the visit, what some of the highlights of the visit were for the visitor, and some strategies and questions teachers may want to reflect upon to make future lessons even better. 8. Are teachers allowed to refuse a visit? No. The purpose of the visits is not to evaluate. The purpose is to provide relevant feedback to teachers in an effort to help them hone their skills and to guide and assess the effectiveness of the supports provided by instructional coaches and supervisors to individual teachers as well as departments, SLCs and buildings. It is important that the visitors have the opportunity to visit all teachers on multiple occasions in order to make the feedback relevant to the individual teacher and to get a full picture of what supports are needed by teachers individually and collectively. 9. Is confidentiality being respected during classroom visits? All visits to classrooms by instructional leaders will remain confidential – no one but the visitor and the teacher will be privy to this information. Data collected by them will be used to help the teacher improve instruction in their classroom. Data will be combined (summarized) across classrooms and teachers to provide information to our technical assistance providers and district staff to improve the supports they provide to teachers and for research purposes.

IRRE, 2007. 46

Administrators 1. How will administrators and supervisors get trained to use this tool and make sure

we’re all on the same page? Following an initial 3-4 hour orientation to the Measuring What Matters project and to the EAR protocol there will be two half-day “calibration” trainings designed by IRRE and supported by district staff – the first will provide instructional leaders opportunities to simulate classroom visits and collectively “calibrate” their responses on the EAR protocol along with experienced users of the protocol and content area experts from district staff. The second will follow multiple classroom visits by vertical teams of instructional leaders from the district and the analysis of the data from those visits – in this training IRRE will provide information from these visits to “recalibrate” classroom visitors’ use of the protocol to ensure that everyone is on the same page in terms of rating engagement, alignment and rigor in different content areas and to discuss how to interpret both individual and aggregated reports coming from the MWM system. The next series of trainings will focus on how instructional leaders in different roles (supervisor, coach, content area leader) can use the EAR protocol and reports to inform their coaching and supervision activities most effectively. 2. How will administrators find the time to use EAR? The primary responsibility of the school administration is to be an instructional leader and foster high academic achievement of students. Therefore, the time spent in classrooms is crucial. This is one process to assess and respond effectively to the needs and growth of teachers and students within your school. 3. What kinds of information will administrators get from EAR? Administrators will have access to the MWM website to access on-line reports of the aggregate results of classroom visits made using the EAR protocol. Summaries of observations conducted by you or those you directly supervise will be available for your use, as will aggregate reports summarizing all observations conducted in your building, SLCs, and departments. EAR reports will examine the extent to which: 1) students are engaged during instructional time; 2) instructional strategies and materials are aligned to district content standards and curricula, and 3) rigorous learning activities and content are evident during instruction. Reports provide information about changes in engagement, alignment and rigor over time for groups of teachers within departments, teaching the same content and from each small learning community.

IRRE, 2007. 47

Students 1. What are these people doing here in my classroom? The purpose of the visits is to provide feedback to teachers in an effort to reinforce positive teaching practices and develop new ideas to make learning more interesting for students. Many adults will be coming into your classroom this year to do this.

Features of EAR Classroom Visit Protocol - Yola€¦ · IRRE, 2010. 1 Measuring What Matters© EAR...

Documents

Transcript of Features of EAR Classroom Visit Protocol - Yola€¦ · IRRE, 2010. 1 Measuring What Matters© EAR...