Read 180

Program Evaluation

FRIT 7738

Spring 2011

Georgia Southern University

April 25, 2011

Executive Summary

The purpose of this evaluation is to determine the effectiveness of the Read 180 program for students

with disabilities at M.D. Roberts middle school by looking at performance data and by interviewing

students in the Read 180 program and the instructors of the program. As a part of the evaluation,

factors for the successful implementation of the program such as the environment of the class and

continued access to a large number of computers will be examined.

The following evaluation questions were reviewed to make the determination:

What is the short-term skill enhancement through review of triennial benchmark assessments

(Star Reading)

What is the Functional skill improvement after completion of the program (beginning and end

Lexile scores).

What is the year over year improvement on standards-based/ Criterion referenced reading

testing (5th and 6th grade CRCT results)

Do the teachers/students feel that the program improves performance? Are there comparable

programs? (Qualitative)

Do teachers/students feel that continued access to large numbers of computers (relative to class

size) is needed?

Multiple data sources were used to review the evaluation questions including CRCT reading and

language arts data for the students’ 5th and 6th grade years, triennial Lexile test scores while the students

were in the program and STAR reading tests as well as interviews of the students currently enrolled in

the program and the instructors of the program. Lexile tests were given to the students at the

beginning, middle and end of the course and the scores were used to set the reading level of the

student in the course. The STAR reading test is a test used by all students in the school and was used as

a comparative test to track the performance of the Read 180 students versus the performance of the

non-program students at the school. Interviews were given to students and instructors to gather

information on the program that would not be readily available to the evaluator through the

quantitative data. The use of surveys was vital to ascertain the perceived importance of the access to

computers and the environment in which the class was held which are both allocated from the overall

resources of our school technology center. The surveys also allowed me to see beyond the standardized

results of the quantitative data.

Review of the data through the program evaluation has found that

31 students have shown an increase in CRCT score in Reading from 2009-2010, though the

increase is small

40 students have improved their functional reading ability (Lexile), though again the relative

gain is small.

17 students have not shown an increase in ability levels as compared to non-program students

100% of students feel that access to computers was somewhat of very vital to their success in

the program

100% of teachers believe that the environment of the class is a vital component to the

curriculum

Through essay response on the questionnaire, several teachers thought that the program was

not implemented fully because of the need for the teachers to focus on CRCT reading/ELA

content.

Recommendations

The results of the review of the data indicate several possible directions for the program and the

continued use of the resources:

Continue to monitor efficacy of intervention through outside means of assessment (AimsWeb)

Research alternative interventions that do not remove students from regular classes and

increase the level of least restrictive environment.

Research alternative methods of implementation for the class due to teacher concerns with

splitting the focus of the program

Continue to make available the current resources that have been made available to the program

from the school technology center

Therefore, it is the recommendation of the evaluation study that Read 180 continue to be the primary

intervention program but that ways for the program to be implemented free from the liabilities of CRCT

testing in ELA. Also, the teachers should be able to focus on implementing the program everyday in the

class and should not be required to implement other curriculums within the program. It is imperative to

continue the access to computers and the current class setting because of the perceived importance by

students and instructors of the use of technology to the success of the program.

Introduction

Purpose

The purpose of this evaluation is to check on the status of the Read 180 reading intervention program at

M.D. Roberts middle school and to evaluate its effectiveness as an intervention tool through the analysis

of quantitative performance data. Secondarily, continued use of part of the Roberts technology center

and devotion of an extensive proportion of equipment to the program will be evaluated through the

performance of the students and the willingness of the students and instructors to change to an

alternate program that uses far less equipment and space.

The evaluation is being conducted because the program has not been evaluated since its beginning 3

years ago and the overall perception of the program by the administration and faculty is mixed and

ambiguous. An evaluation is warranted due to the lack of knowledge on the efficacy of the program and

the fact that the school is devoting a lot of its technological resources to the program. Administrators

feel that if an alternate program could be enacted that has a small need for technological resources,

then the school could allocate some of the technology center’s resources to other intervention

programs.

Audience

The audience of this evaluation is expected to be the administrators at Roberts middle as well as the

instructors of the program. At some point, it should be assumed that the information will be

disseminated to other stakeholders such as the parents of the students in the program.

Limitations of Evaluation

The largest limiting factors for this evaluation was the limited amount of times and resources that were

available to the evaluator. The questionnaires were only given to a sampling of the current students in

the program which caused a small sample size and could not reflect any change or growth in the

implementation of the program over time. The raw data was also limited to one class of students from

the program because data was not taken from the previous 2 groups of students in the program.

Another limitation that occurred in the evaluation was a lack of access to alternate programs that were

being implemented. The county is mainly trying this program right now and I did not have access to

other program’s participants or access to the costs of these programs.

Focus of the Evaluation

Read 180 is a three stage reading intervention program which utilizes various prints and technological

resources. At our school, Read 180 is implemented with students who have failed CRCT reading and is

used in place of the traditional English/ Language Arts class. The program utilizes computer software for

quizzes, data tracking, introductory videos and complementary information. The program also comes

with reading materials (books and workbooks) that are all scaled to a particular Lexile level. Lexile levels

are found for each student through an introductory test and tracked throughout the year with additional

tests. The reading material and workbooks are tailored to the students’ Lexile scores. Lesson plans are

provided to the teacher and strict adherence in implementation is expected.

The program was initiated in the system 4 years ago and has been used as the primary reading

intervention program for middles school since that time. It is being used on approximately 20% of

students in the 6th grade and has been offered on a trial basis to a smaller percentage of 7th grade

students. The program is offered to students that have failed either or both of the Reading or ELA CRCT

tests and is taken in place of the typical ELA class. Currently, the class is given in a 90 minute block but

will be in a 75 minute block due to a scheduling change that will take effect next school year.

Brief Overview of Evaluation Plan and Procedures

Quantitative findings were obtained through the use of the Criterion-Referenced Competency Test in

Reading and ELA, over a 2 year span that included the program students’ 5th and 6th grade test scores.

The CRCT tests are multiple choice, standardized reference testing that is comparative to other students

being taught by the same standards throughout the state. Also used were the Lexile test results for the

students that were given throughout the year.

The Lexile tests are designed to find the relative functional reading level of the students by finding the

appropriate vocabulary for the reading ability of the student. The students then used their Lexile

number to find appropriate reading material. The third quantitative source for information is the STAR

reading test. The STAR test is administered to all students in the school and will produce an Accelerated

Reader reading level for all students. The test consists of a reading passage followed by questions about

the passage. The test results were used as a comparison tool to link the performances of the Read 180

student on the Lexile tests to a test that all students in the school use. Complete results for all 48

students involved in the program are listed in the appendix.

The qualitative data for the evaluation was gathered through a series of student and instructor surveys

given to a sample of the students and all of the instructors. Each student was given 6 questions and

asked to respond not at all, somewhat or very much. These ratings were used instead of a numerical

rating scale to make responses more clear to the evaluator and the questioning easier for the students.

The teachers were given 5 questions with the same ratings system as well as an additional essay format

question so that the teachers could make comments outside of the questionnaire. It was deemed

appropriate to gather some anecdotal details from the instructors to ascertain the overall impression of

the program from the instructors. All of the test questions are included in the appendix.

Evaluation Results/Data

Student survey results were compiled and analysis of the results showed that the most positive results

from the students were from the questions of computer use (11 out of 15 rated very important) and if

the program is improving your overall ability to read (10 out of 15). The importance of computers to the

program seems very strong which is reflected in the students’ responses, but an overall affinity to

technology may skew the results for this question. Although some technological bias may be present in

the results, importance of technology to 100% of the student sample must be noted and reflected in the

recommendations. Secondly, 100% of sample students believed that the program is improving their

reading ability. Again, a bias to personal performance may be influencing this result but it is a strong

indication of the perception of success by the students.

Student Survey Results Not at all

Somewhat Very Much

Do you find the work in the class challenging? 3 5 7Do you believe that the class is improving your reading ability? 0 5 10Would you be interested in trying a new program? 5 5 5How important is it to be able to work on a computer in the class? 0 6 9Do you enjoy the work that you do on the computer? 0 3 11Have you found it easier to read books now that you are in this class? 0 6 9

The results of the teacher survey is somewhat less telling sue to the lack of sample size and close

association of the instructors. The one obvious response to any of the questions is that all of the

instructors rated the importance of the use of computers in the program as very much. The results of

this question mirror the importance of the computers to the program given by the students and present

a strong case for the recommendation of further access to the computers of the school’s Technology

resource center.

The really telling aspect of the teacher questionnaires was the essay response question. All of the

teachers had a huge concern about the effectiveness of the program due to the fact that it is not being

fully implemented due to required preparation session for the CRCT. One teacher stated, “I have not

been implementing the program for 3 weeks because we have been doing CRCT test preparation

questions each day”. Another teacher stated that she had not implemented the program for a month

due to preparation concerns. These concerns are reflected in the ratings by all three instructors rating

the program results as only somewhat prevalent.

Teacher Survey Results Not at all

Somewhat Very Much

Do you feel like the students benefit from this program? 0 3 0Do you see sufficient improvement from most of the students? 0 3 0How important is the use of computers in this program? 0 0 3How important is the environment in which you teach? 0 1 2Would you be likely to want to switch to an alternate program? 1 1 1If you had to guess, how much improvement does the average student experience in the program?

1 2 0

The quantitative data that was reviewed was somewhat superficially deceptive. The vast majority of the

students should improvements in their CRCT result, Lexile number and STAR level but within those

results the average improvement was very small. Another concern was that several students showed a

decrease in their CRCT, Lexile or STAR reading results indicating that the program had been completely

ineffectual as an intervention. The evaluator believes that some of the decrease can be accounted by

irregularities in testing but that the overall appearance of decrease in concerning. The ability of the

program to increase the reading ability in the vast majority of the students show the effectiveness of the

program, but the small level of increase points to a problem in the implementation. Again, the complete

breakdown of results is contained in the appendix.

Quantitative Data SummaryTotal Students # Showed

ImprovementAverage Improvement

5th grade to 6th grade CRCT improvement

40 31 16.8 points

Lexile Growth from August to May

47 40 121.8

STAR Reading 48 31 0.09 points

Improvement

Recommendations

Based upon the evaluation of program data relating to student needs and gains, Technology center

resource use and student/ teacher perceptions this evaluation study concludes:

Program Strengths

An overall improvement of all students’ reading ability through the results of three different

tests

A full incorporation of technology that is viewed as vital to both students and teachers and

brings about a feeling of familiarity to the students

The environment (technology center) is seen as advantageous to the performance of the

students

Students have an overall positive view of the program and would not want to be placed in an

alternate program

Program Weaknesses

Even though the students are showing improvements in the overall reading ability, the level of

improvement is very small and not enough to bring the students up to grade level by completion

of the program.

Teachers feel that they are not given the appropriate time to fully implement the program

For the amount of program participants, the technology center’s resources are being over-

utilized which detracts from other programs

Recommendations

The continued use of the schools technology center’s resource due to the perceived importance

of both teachers and students of computer usage and environment of the class.

The offering of both the Read 180 curriculum and the ELA curriculum to students by

transforming the read 180 class into an elective offering.

Further trials into alternate interventions should take place due to the limited levels of

improvement shown by the Read 180 system. If increased improvement is found to take place

after the implementation of the recommendations, then trials should not be necessary.

Another tracking probe be utilized that can track reading ability by grade level to be able to

illustrate progress more effectively. AimsWeb is one system that can be used.

Appendix A

Read 180 Student Survey

Rating Scale: 1 – Not at all 2 – Somewhat 3 – Very Much

1. Do you find the work in the class challenging? ____

2. Do you believe that the class is improving your reading ability? ____

3. Would you be interested in trying a new program? ____

4. How important is it to be able to work on a computer in the class? ____

5. Do you enjoy the work that you do on the computer? ____

6. Have you found it easier to read books now that you are in this class? ____

Appendix B

Read 180 Teacher Survey

Rating Scale: 1 – Not at all 2 – Somewhat 3 – Very Much

1. Do you feel like the students benefit from this program? ____

2. Do you see sufficient improvement from most of the students? ____

3. If you had to guess, how much improvement does the average student

experience in the program? ____ (months and years, Example 1.5 = 1

grade 5 months)

4. How important is the use of computers in this program? ____

5. How important is the environment in which you teach? ____

6. Would you be likely to want to switch to an alternate program? ____

Appendix C

Quantitative Data Charts

Read 180 Growth Reports

Read 180 Growth Reports

Read 180 Growth Reports

Read 180 Growth Reports