Post on 15-Jul-2015
confidential and proprietary2
Chapter 6 Chapter 7 Chapters 9 & 10Chapters 4 & 5 (group)
and 11-13 (single case)
Figure 1.6. Continuum of constraint levels for different research methods. (Note: The categories of quantitative
and qualitative methods are meant to be for general guidance only since case study and naturalistic and descriptive
research methods can also fall under the category of quantitative research depending on how the data are collected and
analyzed. Also, survey research (Chapter 8), program evaluations (Chapter 14), and action research (Chapter 16) can
also have aspects of both quantitative and qualitative research methods.
confidential and proprietary
Learner Verification Study
15
• Main purpose to provide feedback using pre-publication version of
program.
• Imperfect implementation—full program not completed, starts and stops
throughout, changes in program and technology improvements.
• Even with focus of study, positive results were seen across diverse
settings—non-special education, special education, after-school.
• Social Validation: Teachers and students liked program and reported
effects.
• Statistically significant results seen on TOWRE-2.
• Positive results on GRADE; statistically significant effects for Grade 5
Total and Comp; statistically significant effects for Vocab at Washington
site.
• Increasing trend on DIBELS.
confidential and proprietary16
Background
• Designed by Department of Academic Research as an Exploratory study
• Incorporates 141 students across grades 3 through 8 attending schools in GA, SC, TN, TX, and WA
• All students performing at least two years below grade level in reading
• Demographics– Higher percentage of males (58%) than females – 65% identified as Caucasian– Over half identified with special needs, most of these SLD– 86% free/reduced lunch
confidential and proprietary17
Implementation
• **Study utilized pre-published version of materials **
• Staggered implementation– Late September: SC– October: GA, TN, WA– Early November: TX
• Setting– GA and TX in non-special education classrooms– TN and SC in special education classrooms – WA in an ‘after-school’ setting
confidential and proprietary18
Implementation, cont.
• Consultants conducted on-site visits to monitor implementation throughout the study– Strong implementation for SC, TN, and TX
• All program components; 90 – 120 minutes per day for instruction
– Slight deviations observed in GA• All program components; 90 – 120 minutes per day for instruction
– Partial implementation in WA• Digital and Print; 45 minutes – 1 hour for instruction
– Initial difficulties with technology/connectivity in WA which impacted implementation of the Digital Experience
confidential and proprietary19
Social Validity Questions of Interest• What are teachers’ and students’ attitudes
and perceptions of the program?– Online survey for teachers, end of study
– Student interview, end of the study
• What effect does FLEX Literacy have on students’ perceived reading skill and interest in reading?– Pre- and post-program administration of a 9 item
survey
confidential and proprietary21
Teacher Survey
• All teachers (100%) reported that students were engaged during the Print Experience , Digital Experience, and the Project Experience.
• All teachers (100%) reported that routines were easy to follow, students felt successful, and content standards were addressed in the Print Experience.
• All teachers (100%) indicated that the Teacher’sGuide is well-organized, students felt successful, and content standards are addressed in the Project Experience.
confidential and proprietary22
Teacher Survey, cont.
• Responses on items addressing the Digital Experience varied across sites and largely reflect issues with technology/connectivity that have since been corrected.
• Despite this, the majority of teachers cited the Digital Experience and the opportunity for students to work at their own pace as particular strengths of SRA FLEX Literacy.
confidential and proprietary23
• What four words would you use to describe SRA FLEX Literacy?
• Individual• Challenging• High-expectations• Collaborative• Intense• Flexible• Thought provoking• Life time skills• Interesting• Relevant• Deliberate• Engaging• A very different format
confidential and proprietary24
• 100% of teachers indicated they would recommend SRA FLEX Literacyto a colleague.
• Yes. I already have. Teachers are looking for an individualized program for students struggling in reading and this program (mainly the digital) is equivalent to a student who is sitting one on one with a reading tutor. The teacher knows exactly where the student is and 30 students can work at once on their level, rather than hiring a tutor for 30 minutes of small group once or twice a week. More students benefit from individualized instruction resulting in less being labeled with a disability.
confidential and proprietary26
How did SRA FLEX Literacy help you become a better reader?
• 87% of students reported that they felt SRA FLEX Literacy helped them become a better reader.
0
10
20
30
40
50
60
70
80
Read/PronounceWords
Read Aloud Asks Questions Other
confidential and proprietary27
90% of students’ responses were positive, indicating they enjoyed the program or thought it improved their reading skills.
• What would you tell other students about SRA FLEX Literacy?
0
10
20
30
40
50
60
Fun/Cool Read Better Easy/Hard DK
confidential and proprietary28
59%16%
7%
6%
12%
Digital Experience
Print Experience
Project Experience
Learning
DK
What is the best thing about SRA FLEX Literacy?
confidential and proprietary29
• The majority of students responded that they enjoyed the Project Experience.
– Roughly one-third reported that their favorite part was “working in teams”
– Other responses varied considerably, but included “researching,” “coloring/creating,” “making PowerPoint slides,” and simply, “all of it.”
confidential and proprietary31
• Short, nine-item survey administered before and after participation in FLEX
• Designed to capture student perceived reading skill and interest in reading
– I am a good reader
– I like to read at home
• Response categories ranged from Yes! to Not At All
confidential and proprietary32
All Students
• After participation in FLEX, more students selected Yes!, or Mostly in responding to the statements:
– I am a good reader (15 % increase)
– When reading, I can easily read all of the words (14% increase)
– When reading, I know what all of the words mean (10% increase)
confidential and proprietary33
By Subgroup
• I am a good reader– Grade 5 (25% increase)
• I can read as well as other students– Grade 5 (8% increase)
• When reading, I know what all of the words mean– Grade 5 (20% increase)
• I would enjoy receiving a book as a present– Grade 8 (25% increase)
• When reading, I can easily read all of the words– Grade 8 (33% increase)
confidential and proprietary35
Questions of Interest, cont.
• Association with basic reading, vocabulary, and comprehension skills
– Pretest and Posttest using published, norm-referenced tests
• Association with oral reading fluency skills
– Repeated administrations of the Oral Reading Fluency (ORF) subtest of the DIBELS
confidential and proprietary36
Quantitative Measures
• Test of Word Reading Efficiency, Second Edition (TOWRE-2)– 2 subtests: Sight Word Efficiency and Phonemic Decoding
Efficiency
• Group Reading Assessment and Diagnostic Evaluation (GRADE)– Comprehension and Vocabulary Composites along with a
Total Score
• Oral Reading Fluency (ORF) subtest from the DIBELS– One-minute reading assessment of reading fluency (words
per minute)
confidential and proprietary37
TOWRE-2 and GRADE
• Group administered• Contain parallel forms for pretest and posttest• High reliability/validity• Give norms for off-grade level assessment• Provide standard scores/are norm-referenced
– Allow for comparison across various tests and groups– Can be manipulated for use in statistical tests– Easily interpretable – Often utilized in the identification of students with
disabilities
confidential and proprietary38
A word about “statistical significance”
• Indicates whether or not there is a difference between groups simply due to chance
• Is impacted in part by sample size
– If treatment effects are small (and present), larger samples are more powerful in detecting them.
– With smaller samples, treatment effects (if present) must be relatively larger to be detected.
confidential and proprietary39
…and Effect Size
• Describes the magnitude of the difference• Represents the average difference between groups,
expressed in standard deviation units – For example, an effect size of .30 means “three-tenths of a
standard deviation”
• Effect size is descriptive for that particular group• Cohen’s d
– ‘Small’: .20– ‘Medium’: .50– ‘Large’: .80 and greater– For educational research, .25 is considered minimal level
confidential and proprietary40
Learner Verification
Study (continued)• Effect sizes are measures of magnitude—i.e., how large are the effects.
• Effect sizes were positive. Note that .25 effect size is the minimal level for
educational significance (Adams & Engelmann, 1996); effect sizes over
1.00 are relatively rare in education.
Some example results:
Grade 5
Effect size = .42 on GRADE Comp
Effect size = .38 on GRADE Total
Grade 8
Effect size = 1.49 on TOWRE-2
Tennessee
Effect size = 1.37 on TOWRE-2
Washington
Effect size = .73 on TOWRE-2
Effect size = .52 on GRADE Vocab
confidential and proprietary41
All Students
• Statistical significance achieved for TOWRE-2, effect size is small.
• Non-statistically significant increases for the GRADE.
• For all measures, pretest and posttest scores fall within the Low Average range.
confidential and proprietary42
Student Performance by Grade Level
• Grades 3, 4, 6, and 7– Increases noted
• Grade 5– Statistically significant increases for GRADE
Comprehension (d = .42, small) and GRADE Total (d = .38, small effect).
• Grade 8– Statistically significant difference for TOWRE-2 (d =
1.49, large effect)
confidential and proprietary43
Grade 5
• Statistical significance achieved for GRADE Total and GRADE Comp; effect size is small.
• Non-statistically significant increases for the TOWRE-2 and GRADE Vocab.
• The average comprehension score increased from the Low range to the Low Average range.
confidential and proprietary44
Grade 8
• Statistical significance achieved for TOWRE-2,; effect size is large.
• Stable performance for the GRADE.
• The average TOWRE-2 score increased from the Low Average range to the Average range.
confidential and proprietary45
Student Performance By Site
• Increases shown in Georgia, South Carolina, and Texas
• Tennessee– Statistically significant increases for the TOWRE-2 (d =
1.37, large effect)
• Washington– Statistically significant increases for the TOWRE-2 (d =
.73, medium effect) and GRADE Vocabulary (d = .52, medium effect).
confidential and proprietary46
Tennessee
• Statistical significance achieved for TOWRE-2; effect size is large.
• Non-statistically significant increases for the GRADE.
• The average TOWRE-2 score increased from the Low Average range to the Average range.
confidential and proprietary47
Washington
• Statistical significance achieved for TOWRE-2; effect size is medium.
• Statistical significance achieved for GRADE Vocabulary; effect size is medium.
confidential and proprietary48
SLD Category
• Statistical significance achieved for the TOWRE-2, effect size is large.
• Non-statistically significant increases are noted on the GRADE Total and GRADE Comprehension.
• Stable performance on GRADE Vocabulary.
• The average TOWRE-2 score increased from the Low range to the Low Average range.
confidential and proprietary50
Oral Reading Fluency
• General upward trend across grade levels
• The average gain in words per minute by grade level ranged from a 4% to a 59% increase
• Strongest results for grade 5
confidential and proprietary51
GRADE 5ORF
• At baseline, students read an average of 63 words per minute.
• Students read an average of 100 words per minute on the Spring benchmark.
• Students gained, on average, 37 words per minute. This reflects a 59% increase over baseline levels.
confidential and proprietary55
Because the total number of lessons students receive may have an effect on their scores, the correlation between Lexile® growth and percentage of lessons completed for the program was calculated.
The correlation between Lexile® growth and percentage of yearly lessons completed for the program was statistically significant, r = .559, n = 69, p < .001 (two-tailed). This correlation indicates there was a statistically significant, positive relationship between Lexile® growth and the percentage of yearly lessonscompleted.
confidential and proprietary56
Summary
• The majority of students and teachers reported that they enjoyed using SRA FLEX Literacy, would recommend it to others, and felt successful while using it.
• There were some positive shifts in perceived reading skill and interest in reading, for several groups.
• Pre-post program differences on the TOWRE-2 were determined significant for several groups.
• Oral Reading Fluency scores show an upward trend, over time.
confidential and proprietary
Washington School District Study
57
• Large study in Eastern Washington School District.
• 55%-75% free and reduced price lunch.
• Primary purpose to determine student academic gains.
• Full program implemented for entire year.
• Up to 150 students across Grades 6, 7, and 8.
• Intervention students.
• Multiple measures including Washington State MSP (Measurements of
Student Progress) test results, AIMSWeb, TOWRE-2, GRADE, in-program
assessments.
• Possible math test results on MSP.
• Compare with students who are not in intervention program but with similar
demographics.