Virginia Middle School Research Alliance (VMSRA)Using the early warning system process to identify and

meet middle schoolers needs

Aimee EvanVMSRA Lead, REL Appalachia

Virginia School-University PartnershipSummer Leadership Conference

Charlottesville, VAJune 25, 2014

Introductions

• Name• Role• Division• Previous involvement with a Regional Educational Laboratory (REL)• What brought you to this session

2

Background on REL Appalachia

Regional Educational Laboratory (REL) Program

• U.S. Department of Education, Institute of Education Sciences.• RELs provide regional support for:

– Applied research and evaluation.

– Technical support and information sharing to build capacity to use data for improved education outcomes.

4

REL Appalachia’s Mission

• Meet the applied research and technical support needs of Kentucky, Tennessee, Virginia, and West Virginia.

• Bring evidence-based information to policymakers and practitioners:– Provide support for a more evidence-reliant education system.

– Inform policy and practice for states, districts, schools, and other stakeholders.

– Focus on high-priority, discrete issues and build a body of knowledge over time.

5

What Does REL Appalachia Do?

6

• Assess regional research needs by monitoring emerging education issues and challenges.

• Maintain and refine research alliances through ongoing dialogue between educators in each region and researchers.

• Provide analytic technical support to increase use of data and analysis to understand policies and programs, make decisions, and support effective practice.

• Conduct research and evaluation studies of rigor and method appropriate to the questions the studies attempt to answer.

• Distribute results of REL research across the region.

• Coordinate and partner with other RELs and federal, state, and local education research and technical assistance organizations.

Research Alliances

• What is a research alliance?– A partnership between education stakeholders and REL Appalachia.

• What is the purpose of a research alliance?– As partners, REL Appalachia and alliance members develop and carry out a

research and analytic technical assistance agenda on priority topics.

• Who are the education stakeholders in an alliance?– May include representatives from one or more schools, divisions, state

education agencies, and other organizations (e.g., colleges and universities).

7

VMSRA Goals

9

• Superintendents, assistant superintendents, directors of curriculum and instruction, directors of assessment and testing, principals, teachers from the following divisions:

• Executive directors of SURN (School-University Research Network) and VSUP (Virginia School-University Partnership).

• Virginia Department of Education.• REL Appalachia researchers in rural education, early warning systems

(EWSs), data use.

What Is the VMSRA?

– Louisa County– Norton City– Page County– Pulaski County– Russell County

– Rockingham County– Smyth County– Staunton City– Tazewell County– Wythe County

VMSRA Goals

• Assist middle school practitioners in using data to inform instructional decisionmaking and improve student outcomes by:

– Identifying struggling students who need additional support.

– Selecting, implementing, and monitoring interventions to support students.

• REL Appalachia’s work with the alliance will focus on expanding the state’s EWS to middle schools.

• EWS implementation is taking place beginning with four interrelated projects in 2014/15.

10

What Is an EWS?

• Provides a systematic way to identify (“flag”) students early who are at risk of failure.

• Grounded in research.• Relies on readily available (and familiar) data: ABCs – Attendance,

Behavior, and Course grades/assessment results.• Provides information that is actionable by educators and other adults in

schools and divisions.• Requires educators to diagnose further student needs, and to use

professional judgment to support at-risk students.• Targets resources to support at-risk students while they are still in school,

before they go too far down the road of academic failure and drop out.• Examines patterns and identifies school climate issues.

11

Why Middle School Matters

• A student’s decision to drop out of school is a gradual process that starts well before high school.*– The habits and patterns set in middle school follow students to high school.

– Only 50% of students entering high school two or more years behind grade level in math/literacy are promoted to grade 10.

– Students repeating grade 9 have a 20% likelihood of graduating from high school.

• Students at risk of dropping out can be identified as early as grade 6 and share characteristics of low attendance rates, course failure, and low assessment scores.**

• An EWS helps catch these students before it’s too late.

* Allensworth & Easton (2005, 2007)** Balfanz (2009); Balfanz, R., & Herzog, L. (2005) 12

How Does an EWS Work?

The EWS Continuous Improvement Cycle

Step 1: Establish roles and

responsibilities

Step 2: Use the EWS Middle Grades Tool

Step 3: Review the EWS data

Step 4: Interpret the EWS data

Step 5: Assign and provide

interventions

Step 6: Monitor students and interventions

Step 7: Evaluate and refine the EWIMS

process

14 * Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M., (2013)

Infrastructure and Support is Just as Important as the Process

EWS – A Systems Perspective

Step 1: Establish roles and

responsibilities

Step 2: Use the EWS Middle Grades Tool

Step 3: Review the EWS data

Step 4: Interpret the EWS data

Step 5: Assign and provide

interventions

Step 6: Monitor students and interventions

Step 7: Evaluate and refine the

EWIMS process

16 * Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M., (2013)

A. Data and Computer Data Systems

B. Educator Knowledge & Skills for Data Use

C. School Organization for Data Use

Projects Toward EWS Implementation

VMSRA Projects

• Data catalog of assessments and interventions currently being collected in schools.

• Document how data are currently being used in schools, and what supports and barriers are in place to help or that hinder use.

• Identify most powerful data to track.* • Provide workshops on using data efficiently and effectively to:*

– Identify and target resources to struggling students.

– Monitor each student’s progress toward mastery.

* These are proposed activities and not yet approved by IES. 18

19

What Have We Learned So Far?

• Most data are available for an EWS; using data is difficult. – Extracting or “triangulating” multiple data points is complex.

– Data are not always centrally located in one program.

– Not all pertinent data are electronic; some are kept in hard copy only.

• Analyzing historical patterns for students who are struggling may not be possible because some data do not “save” from year to year.

• Sharing information about students across school levels—from elementary to middle to high school—is not common.

• Assessment data is not always teacher-friendly: working with Alliance to balance assessments FOR learning with assessments OF learning.

Difference Between Assessments OF and FOR Learning

Criteria Assessments OF Learning Assessments FOR Learning

Purpose Check achievement status for public reporting

Help students learn more

Informs Others about students: teachers, school leaders, policy makers, etc.

Students about themselves

Focus on State or local standards Achievement targets that underpin standards

Answers questions

Have students MET standards? Have students mastered the learning blocks/foundations of the learning standards?

Timing After learning is supposed to have occurred

During learning, while there is still time to help

Uses Certification; public reporting; grading; gatekeeper decisions

Understand success; diagnose needs; see progress

20 Adapted from Stiggins, R. http://www.youtube.com/watch?v=EDVHuHyCGmg

Difference Between Assessments OF and FOR Learning

Criteria Assessments OF Learning Assessments FOR Learning

Teachers role

Timekeeper Partner; coach

Students role

Get the highest score; avoid failure Understand success; do better next time

Motivation Reward and punishment Belief that success is within reach if they keep trying

21 Adapted from Stiggins, R. http://www.youtube.com/watch?v=EDVHuHyCGmg

Importance of a Balanced Assessment Portfolio to an EWS

• An EWS uses historical patterns to identify students in need of further assistance.

• It’s up to educators to get students in need of assistance back on track.• Educators need the right data.

– To meet student’s needs, educators have to know what those needs are.

– To determine whether interventions are working, educators need to know what to monitor the progress of, and they need data to be able to do the monitoring.

• Assessments OF learning help determine whether a student is struggling.• Assessments FOR learning help determine how to help the student.

22

23

Discussion

• What is the one thing we must get right to make an EWS worth undertaking?

• What are current hurdles or barriers that would make implementing an EWS difficult? Acting on data difficult?

• Where does an EWS help meet your goals in your division? Where does it fall short?

24

Future Work with the VMSRA

• Data-use interviews: How are you using data? What supports exist? What barriers hinder?

• Validation study*: What are the most powerful data to identify struggling students?

• Workshop for data leaders* (July 23): superintendents, assistant superintendents, curriculum specialists, directors of assessment/accountability, building principals.– What does effective data practice look like?

– How do you create an entrenched strategy around using data, from the division to the classroom? What foundational elements have been found to support effective data use? What additional/different elements are needed to use data effectively?

• Workshops* to assist in creating or building on these foundational elements.

* These are proposed activities and not yet approved by IES.

Alliance Membership Review

• Open to any Virginia middle school, district, or state officer:– Principals and central office staff are greatly encouraged to join.

• Choose a membership level (core or general) at your discretion; time commitment varies:– Core group will participate in monthly phone conferences with REL Appalachia

to develop the research agenda and project implementation plans for 2013/14. Thereafter, calls will be bi-monthly to discuss project progress, receive information, or offer feedback.

– General members will participate in quarterly meetings to receive information and offer feedback on alliance projects.

• All members may choose to participate on alliance project teams or advisory committees.

• REL Appalachia staff will coordinate and manage all REL work.

26

References and Materials

• Allensworth, E. M., & Easton, J. Q. (2005, June). The on-track indicator as a predictor of high school graduation. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf

• Allensworth, E. M., & Easton, J. Q. (2007, July). What matters for staying on-track and graduating in Chicago public high schools: A close look at course grades, failures, and attendance in the freshman year. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/sites/default/files/publications/07%20What%20Matters%20Final.pdf

• Balfanz, R. (2009). Putting middle grades students on the graduation path: A policy and practice brief. Baltimore: Johns Hopkins University, Everyone Graduates Center. Retrieved from http://new.every1graduates.org/putting-middle-grades-students-on-the-graduation-path-a-policy-and-practice-brief/

• Balfanz, R., & Herzog, L. (2005). Keeping middle grades students on-track to graduation: Initial analysis and implications. Presentation at the second Regional Middle Grades Symposium, Philadelphia, PA.

• Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M. (2013). Middle grades early warning intervention monitoring system implementation guide. Washington, DC: National High School Center at American Institutes for Research. Retrieved from http://betterhighschools.org/EWS_middle.asp

Connect with Us!

www.relappalachia.org

@REL_Appalachia

Aimee Evanaevan@QRA-LLC.net

703-655-3695

27