LEXIA RESEARCH

www.lexialearning.com

Efficacy of the Lexia® PowerUp Literacy® Program: State Test Score Improvements for Struggling and Non-Proficient Readers

RESEARCH BRIEF

Lisa B. Hurwitz, PhDresearch@lexialearning.com

Efficacy of the Lexia® PowerUp Literacy® Program: State Test Score Improvements for Struggling

and Non-Proficient Readers

Key Findings

Lexia Research BriefLisa B. Hurwitz, PhD (research@lexialearning.com)

This evaluation of PowerUp is a gold standard randomized control trial (RCT) that meets ESSA’s standards for STRONG research - the highest tier

of efficacy evidence outlined by federal law.

ESSA STRONG LEVEL

PowerUp users gained on average more than 20 percentile points more than non-users on their end-of-year state assessment.

20+ POINTS

Results show that PowerUp is nearly 5x as effective as the average middle school reading intervention.5X

The study took place in two diverse Title I schools.

www.lexialearning.com

BackgroundBy the time they complete middle school, only about 30% of American middle school students are able to read proficiently.1 Over the last few decades, educational policymakers have required students to complete end-of-year summative assessments to quantify proficiency rates within individual school communities and, if necessary, track progress towards increased proficiency. This presents a challenge for educators: Some students begin the academic year reading well below grade level, and it can be quite an undertaking to remediate their individual skill gaps while at the same time preparing them for high-stakes assessments, which focus on grade level skills.2

The Lexia® PowerUp Literacy® blended learning program (PowerUp) was designed to build reading proficiency among struggling and non-proficient secondary readers by targeting common skill deficits. PowerUp provides a systematic sequence of learning activities students work through independently online, integrated with offline teacher-led lessons and paper and pencil practice and generalization activities. The online and offline content is organized into three different instructional strands: Word Study, Grammar, and Comprehension. PowerUp addresses the needs of a wide range of students through individualized and adaptive learning paths within each instructional strand and the inclusion of engaging game-like elements such as winning streaks and polls.

This study evaluated PowerUp users’ performance on a high-stakes standardized assessment. It qualifies as a STRONG evaluation that meets the highest standards for efficacy evidence as outlined by ESSA.

www.lexialearning.com

1 National Assessment of Educational Progress (NAEP). (2019). NAEP report card: Reading. Retrieved from https://www.nationsreportcard.gov/reading/nation/achievement/?grade=8

2 Morgan, H. (2016). Relying on high-stakes standardized tests to evaluate schools and teachers: A bad idea. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 89, 67-72. doi:10.1080/00098655.2016.1156628

MethodSampleThis study took place in a suburban school district just outside Detroit. The district is mid-sized with two middle schools, both of which participated in this study. All of the district’s K-8 buildings receive school-wide Title I funding, and 60% of district students have been identified as economically disadvantaged. Seventy-percent (70%) of middle school students in the district were reading below grade level at the time they entered this partnership with Lexia.

This study included 6 supplemental reading classes for students identified as needing Tier 2 support. Students were selected for these classes based on their grades and standardized test scores, as well as evaluations by school personnel. All students in the district received regular ELA instruction,3 and students in these supplemental classes received an additional 40 minutes of daily reading/literacy instruction.

The teachers of these classes were experienced professionals. Five out of 6 shared personal demographic data for the study. All had Master’s degrees or above, and 3 out of 5 had more than 5 years of teaching experience.

This report focuses on the 38 English-speaking 6th grade students who were enrolled in the supplemental reading classes and completed the state’s end-of-year assessment in the year before and after the intervention.

3 Regular ELA instruction was delivered using a modified version of Houghton Mifflin Harcourt’s Collections ELA program.

STUDENT CHARACTERISTICS

71% FREE/REDUCED LUNCH

100%6TH GRADE TIER 2 READERS 45%

White

47%Black

3% Latinx 5% Multi-racial

www.lexialearning.com

53% MALE

8% SPECIAL ED

ProcedureFour teachers (23 students) were randomly assigned to a treatment group that would use PowerUp in their supplemental reading classes from January through June, and 2 teachers (15 students) were assigned to a control group that would continue to deliver their traditional supplementary curriculum.4,5 The district historically struggled to accelerate students’ reading proficiency and requested that classes be disproportionately assigned to the treatment condition.

End-of-Year Summative AssessmentStudents in the district take the Michigan Student Test of Educational Progress (M-STEP) English Language Arts (ELA) assessment as their end-of-year summative test. M-STEP is a computer-adaptive assessment with passage-based writing prompts that students complete over 2 hours. It measures achievement in reading, writing, listening, and research. In the present study, 5th grade scores from before the intervention were treated as pre-test scores, and 6th grade scores from after the intervention were considered post-test scores. Fifth grade students receive scaled scores that can range from 1409–1550, and 6th grade students receive scaled scores that can range from 1508–1655.

ResultsPowerUp ImplementationOn average, PowerUp students used the program for 15 weeks between January and June of the research study year, completing an average of 64 minutes of online work per week.

4 For their traditional curriculum, the district used McGraw-Hill’s Corrective Reading or Read to Achieve: Comprehending Content Area Text. Corrective Reading is an intervention that provides instruction through the 7th grade level. Read to Achieve is designed for middle and high school students and features social studies and science texts. Both programs focus primarily on comprehension. The lowest level of Corrective Reading used by a minority of students in the district also provides instruction in decoding and phonemic awareness.

5 Condition assignment was stratified by school. As such, each middle school contained one control and two treatment classes.

www.lexialearning.com

STUDY DURATION

• 15 weeks • 64 min/week

End-of-Year Summative Assessment OutcomesIn the year prior to the study, when students were in 5th grade, those ultimately enrolled in PowerUp classes had similar scores to those in the control group who used the traditional curriculum. However, after the intervention, students in the PowerUp classes averaged more than 6 points higher on the M-STEP than students in the control group.

The difference in scaled score points translates to a large effect. Researchers use the effect size metric Cohen’s d to quantify the impact of an intervention. According to a review by the US Department of Education, the average reading intervention for middle schoolers has an effect size of Cohen’s d = .11.6 Cohen’s d on the M-STEP for these students was .54, or nearly 5X THE IMPACT OF THE AVERAGE READING INTERVENTION for middle school students. That is the equivalent of a boost of over 20 percentile points.7

See the technical appendix for more information on the calculation of these results.

6 Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, Institute of Educational Sciences, US Department of Education.

7 What Works Clearinghouse. (2014). Procedures and standards handbook (3rd ed.). Washington, DC: What Works Clearinghouse, Institute of Education Sciences, US Department of Education.

www.lexialearning.com

Assessment Scores

PowerUpTraditional Curriculum

6.56POINTS

0.41POINTS

Pre-Intervention Post-Intervention

1550

1600

1500

1450

POWERUP WAS NEARLY

5X AS EFFECTIVE AS THE AVERAGE MIDDLE

SCHOOL READING INTERVENTION

ConclusionPowerUp succeeded in promoting reading achievement for 6th grade students identified as needing Tier 2 intervention. PowerUp students in this study scored significantly higher on M-STEP than control students using the traditional curriculum. Given that PowerUp students used the program for only 15 weeks, these findings are particularly remarkable.

It is likely that PowerUp students entered high-stakes testing at the end of the study with a more robust toolkit of literacy skills than their peers in the control group. Based on their experience in PowerUp, they may have been more confident and motivated to read and apply their literacy skills. It should be recognized that the activities in PowerUp mainly target word study, grammar, and comprehension skills in the context of reading, while M-STEP assesses passage-level skills with an emphasis on writing. Therefore to succeed on M-STEP, PowerUp students needed to not only master content in PowerUp, but also generalize their learning to the assessment items on M-STEP. PowerUp students were able to consolidate and transfer their skills, producing notably stronger scores on M-STEP than control students.

Ultimately, these findings provide strong evidence of PowerUp’s effectiveness with struggling and non-proficient middle school readers. The impact of PowerUp may continue to grow as students engage with the program more deeply over a longer period of time.

www.lexialearning.com

© 2020 Lexia Learning, a Rosetta Stone company. Lexia®, Core5®, and other trademarks, names, and logos used herein are the property of Rosetta Stone Ltd. and/or its subsidiaries, and are registered and/or used in the United States and other countries. Additional trademarks included herein are the property of their respective owners.

www.lexialearning.com

Technical AppendixA multi-level model was calculated using HLM 8 to test for differences in post-intervention M-STEP scores between the PowerUp and control groups. The mixed model took the following form:

Postij = Υ00 + Υ01 * Treatmentj + Υ10 * Special Edij + Υ20 * Preij + u2j * Preij + rij ,

where: • Postij is the post-intervention M-STEP score for student i in class j• Υ00 is the expected post-intervention M-STEP score for the average student in a control class• Υ01 is an estimate of the difference in expected post-intervention M-STEP scores between students in treatment and control classes• Treatmentj is an indicator of being in a treatment class (1 = treatment, 0 = control) • Υ10 is an estimate of the difference in expected post-intervention M-STEP scores between general and special education students • Special Edij is an indicator of having a special education designation (1 = special education, 0 = general education)• Υ20 is an estimate of the association between pre-intervention and post-intervention M-STEP scores• Preij is each student’s grand-mean centered pre-intervention M-STEP score• u2j is a class-level residual slope term, and • rij is a student-level residual term.

We controlled for additional demographic variables in alternate models but omitted them from the final model because their fixed effects were not significant.

The slope on pre-intervention M-STEP scores was modeled as random. It was decided to do so because the variance component was significant (0.13, Χ2 (5, N= 38) = 15.34, p = .009), indicating that the impact of prior reading achievement varied between students within classes. In alternate models, it was found that the variance components for the other predictors were not significant, and, as such, they were modeled as fixed.

Results of this model revealed that there was a positive and statistically significant difference between the PowerUp and control classes, Υ01 = 6.39, SE = 2.99, p = .041.