THE EFFECTIVENESS OF A SOCIAL-EMOTIONAL LEARNING …Students in the first treatment group received...
182
THE EFFECTIVENESS OF A SOCIAL-EMOTIONAL LEARNING PROGRAM WITH MIDDLE SCHOOL STUDENTS IN A GENERAL EDUCATION SETTING AND THE IMPACT OF CONSULTATION SUPPORT USING PERFORMANCE FEEDBACK by BARBARA ANN GUELDNER A DISSERTATION Presented to the College of Education and the Graduate School of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Philosophy June 2007
Transcript of THE EFFECTIVENESS OF A SOCIAL-EMOTIONAL LEARNING …Students in the first treatment group received...
THE EFFECTIVENESS OF A SOCIAL-EMOTIONAL LEARNING PROGRAM WITH
MIDDLE SCHOOL STUDENTS IN A GENERAL EDUCATION SETTING AND THE
IMPACT OF CONSULTATION SUPPORT USING PERFORMANCE FEEDBACK
by
BARBARA ANN GUELDNER
A DISSERTATION
Presented to the College of Education and the Graduate School of the University of Oregon
in partial fulfillment of the requirements for the degree of
Doctor of Philosophy
June 2007
ii “An Investigation of the Effectiveness of a Social-Emotional Learning Program With
Middle School Students in a General Education Setting and the Impact of Consultation
Support Using Performance Feedback,” a dissertation prepared by Barbara Ann Gueldner
in partial fulfillment of the requirements for the Doctor of Philosophy degree in the
College of Education. This dissertation has been approved and accepted by:
____________________________________________________________ Kenneth W. Merrell, Chair of the Examining Committee ________________________________________ Date Committee in Charge: Kenneth W. Merrell, Ph.D., Chair Michael Bullis, Ph.D. K. Brigid Flannery, Ph.D. Lynn Kahle, Ph.D. Accepted by: ____________________________________________________________ Dean of the Graduate School
iii
© 2006 Barbara Ann Gueldner
iv
An Abstract of the Dissertation of Barbara Ann Gueldner for the degree of Doctor of Philosophy in the College of Education to be taken June 2007 Title: AN INVESTIGATION OF THE EFFECTIVENESS OF A SOCIAL-
EMOTIONAL LEARNING PROGRAM WITH MIDDLE SCHOOL STUDENTS
IN A GENERAL EDUCATION SETTING AND THE IMPACT OF
CONSULTATION SUPPORT USING PERFORMANCE FEEDBACK
Approved: _______________________________________________
Kenneth W. Merrell, Ph.D.
This dissertation study investigated the efficacy of the Strong Kids curriculum, a
promising social and emotional learning program developed from research-based
resiliency intervention tools and evidence-based instructional models for enhancing
children’s assets and mental health. Although initial pilot efforts have been encouraging,
additional evidence is needed regarding its efficacy. Three groups of sixth grade students
in a general education setting were studied. Students in the first treatment group received
Strong Kids and the teacher received intensive consultation support. The students in the
second treatment group also received Strong Kids, but the teacher did not receive
additional consultation support. The third group was comprised of students in a control
group. A unique feature of this study was the use of consultation support that utilized
v performance feedback with principles of motivational interviewing. Teachers’ attitudes
toward the curriculum were also evaluated. Results of this study indicated a statistically
significant and large increase in knowledge of social-emotional concepts and coping
skills for the treatment groups, as evidenced by statistically significant gains from pretest
to posttest, as well as by statistically significant differences between the treatment and
control groups on these variables at posttest. Internalizing symptoms were generally not
affected by the intervention, nor were office disciplinary referrals with the exception of
minor infractions. Overall, teachers had a positive attitude toward social-emotional
learning and the curriculum, indicating a relatively strong degree of social validity for the
Strong Kids curriculum. Implications of this study for educational practice, continued
refinement of the Strong Kids programs, and future research efforts in this area are
discussed.
vi
CURRICULUM VITAE
DATE OF BIRTH: March 31, 1970 GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: University of Oregon University of Wisconsin-Whitewater University of Wisconsin-Madison DEGREES AWARDED: Doctor of Philosophy, School Psychology, June, 2007, University of Oregon Master of Science, Education, School Psychology, 1997, University of Wisconsin,
Whitewater Bachelor of Science, Psychology, 1992, University of Wisconsin, Madison AREAS OF SPECIAL INTEREST: Social and Emotional Learning in School Settings Prevention/Early Intervention of Internalizing Problems Parent Training Pediatric Psychology PROFESSIONAL EXPERIENCE: Pediatric Psychology Internship, The Children’s Hospital, Denver, 2006-present- Practicum Student, Child and Family Center, University of Oregon, 2004-2006 Practicum Student, Corvallis School District, Corvallis, Oregon, 2005 Practicum Student, Bethel School District, Eugene, Oregon, 2004 Graduate Teaching Fellowships, University of Oregon, 2003-2006 Supervised College Teaching, University of Oregon, 2003-2006
vii Officer, Association of School Psychology Students, 2005-2006 Committee Member, University of Oregon, 2005 Consultant, Thurston Middle School, Springfield, Oregon, 2005 Consultant, Eddyville Charter School, Eddyville, Oregon, 2004 Research Experience, University of Oregon, Dissertation Project, 2005-2006 Research Experience, University of Oregon, Oregon Resiliency Project, 2003-
present Research Experience, University of Oregon, ECS, 2004-2005 Research Assistant, University of Wisconsin, Psychology Department, 1991-1992 Supervisor for School Psychology Trainees, Springfield School District, 2000-2003 School Psychologist, Springfield School District, Springfield, Oregon, 1999-2006 Member, Crisis Response Team, Springfield School District, 1999-2006 Assessor, Lane County rural school districts’ evaluation program, 2002 Book Reviewer, Loving Your Teenage Daughter (Whether She Likes It or Not),
Debra Whiting Alexander, Ph.D., New Harbinger, 2001 Committee Member, Association of School Psychology Students, 2003-2004 Committee Member, Positive Behavioral Supports, 2001-2002 Committee Member, Springfield School District, Springfield, Oregon, 1999-2000 Committee Member, Springfield School District, Springfield, Oregon, 1996-1998 School Psychologist, Poynette School District, Poynette, Wisconsin, 1997-1999 School Psychology Intern, Poynette School District, Poynette, Wisconsin, 1996-
1997 Practicum Student, Watertown School District, Watertown, Wisconsin, 1995-1996 Wisconsin School for the Deaf, Clinical Rotation, 1995
viii Children’s Counselor, C Lazy U Ranch, Granby, Colorado, 1992-1994 Substitute Teacher, Grand County Schools, Colorado, 1993-1994 Child Care Provider, Winter Park Ski Resort, Winter Park, Colorado, 1992-1993 Volunteer, Grand Co. Domestic Violence Response Team, Colorado, 1994
Volunteer, University of Wisconsin Hospital Pediatric Unit, Madison, Wisconsin, 1991
Volunteer, Teen Parenting Program, Madison, Wisconsin, 1990 GRANTS, AWARDS, HONORS:
Clare Wilkins Chamberlin Memorial Award, $1500, An Investigation of the Effectiveness of a Social-Emotional Learning Program with Middle School Students in a General Education Setting and the Impact of Consultation Support Using Performance Feedback, University of Oregon, 2006 Graduate School Research Award, $500, An Investigation of the Effectiveness of a Social-Emotional Learning Program with Middle School Students in a General Education Setting and the Impact of Consultation Support Using Performance Feedback University of Oregon, 2006
Graduate Teaching Fellowships, University of Oregon, 2003-2006 Song Family School Psychology Scholarship, University of Wisconsin, 1995 PUBLICATIONS: Merrell, K. M., Carrizales, D., Feuerborn, L., Gueldner, B. A., & Tran, O. (in
press). Strong kids: A social and emotional learning curriculum for students in grades 4-8. Baltimore, MD: Brookes Publishing.
Merrell, K. M., Carrizales, D., Feuerborn, L., Gueldner, B. A., & Tran, O. (in
press). Strong teens: A social and emotional learning curriculum for students in grades 9-12. Baltimore, MD: Brooks Publishing.
Merrell, K. M., Gueldner, B. A., Ross, S., & Isava, D. (in press). How effective
are school bullying intervention programs? A meta-analysis of intervention research. Psychology in the Schools.
ix Merrell, K. M., Gueldner, B. A., Tran, O. K. (in press). Social and Emotional
Learning: A School-wide Approach To Intervention for Socialization, Friendship Problems, and More. In B. J. Doll & J. A. Cummings (Eds.), Population-based Services of School Psychologists. Washington DC: National Association of School Psychologists.
x
ACKNOWLEDGEMENTS
I wish to express sincere appreciation to the members of my dissertation
committee, Professors Merrell, Bullis, Flannery, and Kahle for their service and
assistance in the preparation of this manuscript. It is with utmost respect that I wish to
sincerely acknowledge Dr. Kenneth Merrell for his unwavering optimism, dedication,
and support during the completion of this project. Thank you for your supportive
mentoring and the positive, constructive, and collaborative feedback you consistently
provided. Thank you to Springfield School District; in particular, Dr. Keith Hollenbeck,
Cathy Kennedy Paine, Carl Swan, Jody Sedlack, Angela Copeland, Troy Thorsby, Jim
Crist, Elaine Lessar, Sonya Black, and Lynn Landerholm. A special thank you to the
school psychologists in Springfield School District for seven years of collegial support
and inspiration. This dissertation project was supported in part by the Oregon Resiliency
Project, the Clare Wilkins Chamberlin Memorial Award, a Graduate School Research
Award.
Thank you to my family who encouraged me while I pursued a childhood dream
and my wonderful friends whom I dearly appreciate. A special thank you to my parents,
Dr. Terry and Judy Gueldner, for faith, courage, and example. Finally, to Grandma Ardis
Helen Severson Fostvedt, who cheered for me from the sidelines, and to Grandpa Sidney
xi Karl Fostvedt, Nanny Mildred Margaret Ella Herrmann Gueldner Rusch, and Papa, Dr.
Louis Henry Gueldner, who cheered from above.
xii TABLE OF CONTENTS
Chapter Page I. INTRODUCTION ................................................................................................1
Statement of the Problem .................................................................................1 Research Questions ..........................................................................................4
II. LITERATURE REVIEW ....................................................................................6
The Mental Health Needs of Youth Today........................................................6 The Need for Prevention and Early Intervention................................................7 Response of the Federal Government................................................................7 Response of State Government .........................................................................8 Schools Are a Venue for Prevention and Intervention........................................9 Social and Emotional Factors Influence Academic Achievement..................... 10 Enhancing Social and Emotional Competence................................................. 10 Internalizing Problems Co-Morbid With Externalizing Problems..................... 12 Integrating Social and Emotional Concepts Into Academic Curricula............... 13 Standards for Social and Emotional Learning Programs .................................. 14 Primary Competencies Needed to Promote Social and Emotional Learning ..... 15 Teacher Involvement in Social and Emotional Learning.................................. 16 Consultation and Program Implementation ..................................................... 16 The Need for Evaluating the Use of Consultation Support and Social and
Emotional Learning Programming......................................................... 17 Performance Feedback as a Means for Providing Consultation Support ........... 18 Incorporating Motivational Interviewing Into the Consultation Process............ 19 The Value of Measuring Treatment Integrity................................................... 21 The Strong Kids Social and Emotional Learning Curriculum........................... 23 The Need to Study Strong Kids With a General Education Population ............. 24 An Interest in Evaluating Strong Kids in Conjunction With Consultation Support .............................................................................................. 25 The Importance of Evaluating Teachers’ Acceptability of Strong Kids............. 26
III. METHOD ....................................................................................................... 28
Design .......................................................................................................... 28 Participants and Setting .................................................................................. 29 Independent Variables.................................................................................... 31 Dependent Variables and Measures ................................................................ 32 Procedures ..................................................................................................... 38
xiii Chapter Page
IV. RESULTS ....................................................................................................... 44 Analyses…………….………………………………………………………. .. 44
Effect of Strong Kids on Knowledge of Social-Emotional Behavior ................ 46 Effect of Strong Kids on Internalizing Symptoms............................................ 48 Effect of Consultation on Knowledge, Internalizing Symptoms, and Office
Disciplinary Referrals......................................................................... 59 Magnitude of Treatment Effects for Self-report Measures and Disciplinary
Referrals............................................................................................. 66 Social Validity of the Strong Kids Curriculum ................................................ 70 Internal Consistency for Dependent Measures................................................. 72
V. DISCUSSION …………………………………………………………………...75 Summary of Main Findings ............................................................................ 75 Impact on Knowledge of Social-Emotional Concepts and Skills ...................... 76 Impact on Internalizing Symptoms Associated With Emotional-Behavioral
Problems ........................................................................................... 77 Impact of Consultation Support on Knowledge, Symptoms, and Office
Disciplinary Referrals......................................................................... 79 Impact on Office Disciplinary Referrals.......................................................... 82 Teachers’ Attitudes Toward the Strong Kids Curriculum................................. 83 Limitations..................................................................................................... 84 Implications for Future Research .................................................................... 86 Implications for Practice................................................................................. 88 Conclusions ................................................................................................... 91
APPENDICES
A. STRONG KIDS KNOWLEDGE TEST ............................................................. 93 B. CHILDREN’S DEPRESSION INVENTORY ................................................... 98
C. INTERALIZING SYMPTOMS SCALE FOR CHILDREN ............................. 101 D. STRONG KIDS SOCIAL VALIDITY SURVEY ............................................ 104
E. TREATMENT INTEGRITY CHECKLISTS ................................................... 110 F. GRAPH TO TRACK TREATMENT INTEGRITY DATA............................... 123
G. RECRUITMENT LETTER ............................................................................. 125 H. GENERAL INFORMATION HANDOUT ...................................................... 127
I. PARENT CONSENT LETTER, TREATMENT GROUPS................................ 130
xiv J. PARENT CONSENT LETTER, CONTROL GROUP ...................................... 134
K. STUDENT ASSENT, CONTROL GROUP ..................................................... 137 L. STUDENT ASSENT, NO CONSULTATION GROUP ................................... 140
M. STUDENT ASSENT, CONSULTATION GROUP ......................................... 143 N. TEACHER CONSENT.................................................................................... 146
O. TRAINING AGENDA ................................................................................... 150 P. FREQUENTLY ASKED QUESTIONS HANDOUT ....................................... 152
Q. PERFORMANCE FEEDBACK CHECKLIST ................................................ 154 R. PHONE CHECK-IN SHEET .......................................................................... 156
BIBLIOGRAPHY ..................................................................................................... 158
xv LIST OF FIGURES
Figure Page 1. Summary of Treatment Integrity Data Over Time ............................................... 37
2. Knowledge of Social-Emotional Skills ............................................................... 48 3. Internalizing Symptoms as Measured by ISSC Total Mean Raw Scores .............. 51
4. Negative Affect/General Distress Symptoms as Measured by ISSC: Factor 1 Scale ........................................................................................................... 54
5. Positive Affect as Measured by ISSC: Factor 2 Scale .......................................... 57 6. Internalizing Symptoms as Measured by the CDI................................................. 59
7. Mean Total Office Disciplinary Referrals ........................................................... 65
xvi LIST OF TABLES
Table Page 1. Research Design................................................................................................. 28
2. Demographic Information of Student Participants................................................ 30 3. Summary of Group Status .................................................................................. 32
4. Means and Standard Deviations for Self-Report, Quantitative Dependent Measures, Listed for Pretest and Posttest Assessments ................................. 45
5. Mixed 2-Way ANOVA for the Interaction Effect of Group and Time on Knowledge of Social-Emotional Skills as Measured by Strong Kids Knowledge Test .......................................................................................... 47
6. Mixed 2-Way ANOVA for the Effect of Group and Time on Internalizing Symptoms as Measured by ISSC Total Raw Score........................................ 50
7. Mixed 2-Way ANOVA for the Effect of Group and Time on Feelings and Behaviors Associated with Negative Affect/General Distress as Measured by ISSC: Factor 1 Scale .............................................................................. 53
8. Mixed 2-Way ANOVA for the Effect of Group and Time on Positive Affect as Measured by ISSC: Factor 2: Positive Affect Scale....................................... 56
9. Mixed 2-Way ANOVA for Group and Time on Internalizing Symptoms as Measured by the CDI................................................................................... 58
10. Descriptive Statistics for Dependent Measures at Posttest, Comparing the No Consultation Group and the Consultation Group Effect of Strong Kids on Disciplinary Referrals .................................................................................. 61
11. Descriptive Statistics for Office Disciplinary Referral Data................................ 62
12. Mixed 2-Way ANOVA for Group and Time on Total Office Disciplinary Referrals...................................................................................................... 64
13. Magnitude of Effect from Pretest to Posttest for Dependent Measures as Measured by Effect Sizes............................................................................. 68
14. Magnitude of Effect at Posttest Among Treatment Groups for Self-Report Measures’ Total Scores as Measured by Effect Sizes .................................... 70
15. Alpha Coefficients for Quantitative Self-Report Measures at Pretest and Posttest........................................................................................................ 73
16. Correlation Coefficients Depicting Convergent and Discriminant Validity of Dependent Measures.................................................................................... 74
xvii
1
CHAPTER I
INTRODUCTION
This chapter includes a brief overview to frame the mental health needs of youth,
efforts to address current problems and concerns, the promising use of social and
emotional learning programs in the schools, and the rationale and purpose for this
doctoral dissertation research study. The chapter begins with a general statement
regarding the current problems this study addressed as well as the goals this project
intended to achieve. The research questions addressed in this investigation are listed at
the end of this chapter, which provides a framing introduction to the remaining sections
of the dissertation document.
Statement of the Problem
Close attention must be given to the mental health needs of youth to ensure
positive psychosocial development throughout their lifespan. Inadequate or poor
development of social and emotional competencies can lead to acute and chronic
difficulties, ultimately culminating in diminished productivity and overall life
satisfaction. New interventions are being introduced to enhance skill development, but
they must have an evidence-basis to demonstrate their effectiveness (Greenberg, et al.,
2003). Schools are increasingly being used as a community resource to provide students
with these skills; however, teachers are often asked to provide this training with little
2 support and concurrently with increasingly high demands to ensure their students meet
rigorous academic standards.
Children in the 21st century exist in the context of social issues which incessantly
make media headlines. Natural disasters, fragmented families, stressors associated with
poverty and abundance alike, and community and school violence are directly
experienced and observed. When children are inevitably exposed to such stressors, their
ability to successfully cope can be challenged. Many children do not develop effective
skills for coping with their challenges, and the results can range from mildly distracting
to bothersome to tragic. Sadness, irritability, chronic anxiety, panic, and hopelessness as
well as anti-social behavior such as drug use and high-risk behavior can insidiously
manifest or unequivocally appear. Without the skills to cope with life’s inevitable
stressors, children face being at-risk for experiencing these symptoms as well as poor
academic performance, difficulties with interpersonal relationships, and likely modeling
poor coping skills to their children. Similar to children receiving vaccines to prevent
serious illnesses, children should be provided an opportunity to receive psychosocial
inoculations that can promote resistance against the ill-effects procured from life
stressors.
Increasingly, schools are re-examining their educational philosophy and practices,
recognizing not only that academic achievement is inextricably linked to social,
emotional, and behavioral competence (Zins, Bloodworth, Weissberg, & Walberg, 2004),
but that schools can play a vital role in shaping citizens who will contribute to society in
a way that is productive, meaningful, ethical, and healthy (Greenberg, et al., 2003).
Subsequent to embracing this philosophy has been the hope that schools can indeed have
3 such a profound impact and a conviction to bring this hope to fruition through strategic
action.
Students are engaged in school-based curricula to promote the acquisition of
social and emotional skills. Social and emotional learning has become the neologistic
descriptor for this effort and new curricula are being developed with social and emotional
skills development in mind. Professional standards demand these curricula be proven
effective (Greenberg, et al., 2003) and the professionals who use them must have the
support necessary to implement them with accuracy and confidence (Zins, Bloodworth, et
al., 2004).
Strong Kids, a recently developed prevention and early intervention curriculum to
promote children’s mental health and resiliency, has been studied over the past four years
and has shown evidence of successful student outcomes pertaining to social and
emotional learning (Castro-Olivo, 2006; Isava, 2006; Merrell, Juskelis, Tran, &
Buchanan, 2006). Additional evidence is needed regarding the efficacy of this curriculum
in a general education setting, particularly with middle school-age students.
Although recommendations have been made in support of educational
professionals receiving ongoing support when implementing a social and emotional
learning curriculum (CASEL, 2004; Zins, Bloodworth, et al., 2004), there are no studies
that demonstrate whether or not consultation, when used with a social and emotional
learning program, can be an effective means to provide such assistance. Performance
feedback is a technology that could prove effective in this context. Although the process
in which performance feedback is delivered is believed to be collaborative (Noell, et al.,
2005), it may be helpful to consider resistance-to-change issues commonly found when a
4 consultee is asked to change a behavior that may be impeding greater success or
outcomes. Techniques used in motivational interviewing may enrich a consultation
process that is frequently used in educational settings.
This study investigated the use of a social-emotional learning curriculum as a
prevention and early intervention means to provide skills to general education students
and derail current and/or future internalizing problems from perpetuating. This study
expanded upon prior research efforts to determine the evidence base for a promising
social-emotional learning curriculum, Strong Kids, as well as evaluate the use of
consultation support provided to a teacher during implementation. Consultation support
combined techniques used in performance feedback and motivational interviewing to
provide a teacher with feedback on the implementation of the curriculum, as well as
addressing likely resistance-to-change issues. It was hypothesized that students would
benefit from receiving the curriculum as well receiving instruction from a teacher who
was provided ongoing consultation support, and teachers would find the curriculum easy
to use and easily integrated into existing academic curricular activities. The goal of this
study was to expand upon the literature base in support of the use of social-emotional
curricula in a general education setting, to find evidence for a structured approach to
consultation support while addressing naturally occurring resistance-to-change issues
often found in consultees, and find continued support for evaluating teacher acceptability
of a social-emotional program in the school setting.
Research Questions
Given the needs briefly described in this introduction, and the overall rationale for
this project, the current study examined the following research questions:
5 1. What is the effect of participation in the Strong Kids social and emotional
learning curriculum on knowledge of healthy social-emotional behaviors?
2. What is the effect of participation in the Strong Kids social and emotional
learning curriculum on emotional-behavioral problem symptoms among middle
school students in a general education setting?
3. What is the effect of participation in the Strong Kids social and emotional
learning curriculum on office disciplinary referrals among these same students?
4. Are student outcomes improved when teachers are provided regular performance
feedback from a consultant having expertise in Strong Kids?
5. To what extent is the Strong Kids curriculum perceived by teachers to a)
align with their goals and the goals of the curriculum, b) have procedures
which are viewed as acceptable and c) result in desirable outcomes for
students and teachers.
6
CHAPTER II
LITERATURE REVIEW
The literature review conducted for this study is not considered to be exhaustive,
but sufficient to understand the current state of the field, develop research hypotheses
regarding current areas of interest, and determine appropriate research methodology to
conduct this study. A keyword search (using the PSYCHLIT and ERIC, and Google
Scholar databases) was used to conduct this review and included terms such as: social
and emotional learning, internalizing problems and disorders, externalizing problems and
disorders, comorbidity, consultation, performance feedback, motivational interviewing,
treatment integrity, social validity, legislation and social-emotional learning, social,
emotional, and academic competency, prevention science, and interventions.
The Mental Health Needs of Youth Today
An estimated 20% of school-age youth experience mental health problems during
the course of any given year (Coie, Miller-Johnson, & Bagwell, 2000; Greenberg,
Weissberg, O’Brien, et al., 2003). Of the youth experiencing these problems, an
estimated 0.4% to 8.3% of adolescents experience depression (Greenberg, Domitrovich,
& Bumbarger, 2001). Often in crisis, approximately 80% of youth with mental health
problems do not receive effective intervention (Greenberg, et al., 2003). Without
intervention, they may experience a domino effect of calamitous and cumulative
problems across their lifespan. For example, a teen fraught with depression may
experience sadness, poor concentration, irritability, interpersonal conflict, social
7 isolation, and decreased academic performance. Left undetected and untreated,
academic failure, school drop out, joblessness, poverty, conflicted interpersonal
relationships, and sometimes suicide needlessly result (Michael & Crowley, 2001).
Billions of dollars are annually spent in the United States on treatment for existing mental
disorders, lost productivity, mortality and criminal justice costs (Coie, Miller-Johnson, &
Bagwell, 2000). Overwhelmingly, our society pays a tremendous social, emotional, and
monetary price for these mental health problems.
The Need for Prevention and Early Intervention
Coie, et al. (2000) emphasized the urgency with which prevention programs are
needed to change the negative trajectory of mental health problems among children and
youth. The cost of providing treatment for existing and chronic mental health problems
far exceeds the cost of providing prevention programs which may deter problems from
even occurring in the first place. We also face a future shortage of mental health
professionals available to provide treatment to those in need (Coie, et al.). Many
individuals who need help simply may not get it. Perhaps most importantly is a moral and
ethical obligation for providing effective, evidence-based prevention information to all
who face life’s challenges. Although programs designed to prevent negative outcomes
seem promising in clinical settings (Michael & Crowley, 2001), more research is needed
regarding the effectiveness of prevention and intervention methods used in the schools
for youth experiencing depression symptoms.
Response of Federal Government
The need for prevention and intervention efforts in the United States has not gone
unnoticed by elected officials in government. In the summer of 2004, the United States
8 Congress passed a bill which specifically addressed the seriousness and urgency in
which children and young adults need prevention and intervention services
(http://gsmith.senate.gov). Following the tragic death of his son by suicide, Oregon
Senator Gordon Smith led an effort to prevent youth suicide by advocating for
prevention, early intervention, and treatment services to school and college-age youth.
This legislation was the Garrett Lee Smith Memorial Act, which resulted in the creation
of a program within the Substance Abuse and Mental Health Services Administration that
could facilitate the development of statewide prevention and intervention strategies
delivered to community agencies and in particular, college campuses. Although this act
and subsequent program highlights perhaps one of the most deleterious outcomes of
children’s mental health difficulties, namely suicide, it has raised tremendous awareness
regarding the mental health concerns of youth and the need for a community-based
response to transform sympathetic and passive observation of the problem as well as
perfunctory responses into informed, organized, and resourceful prevention and
intervention efforts.
Response of State Government
The State of Illinois recently took similar action in an effort to address children’s
mental health needs in an organized and systematic fashion (https://www.ivpa.org). In
2003, the Children’s Mental Health Act was passed which initiated the formation of the
Illinois Children’s Mental Health Partnership. This group developed a blueprint that
outlined multiple areas of need throughout Illinois and strategically matched them with
recommendations for appropriate action. This blueprint represented a coordinated
community effort in identifying prevention, early intervention, and treatment resources
9 for the sole purpose of providing youth from birth to age 21 comprehensive mental
healthcare. The public school system was specifically identified as a primary community
resource from which youth may receive social and emotional education and support.
More recently, in 2006, the State of New York followed the lead of Illinois and enacted
new children’s mental health legislation promoting the use of social-emotional learning
programs in schools.
Schools Are a Venue for Prevention and Intervention
Indeed, schools are an excellent venue for providing prevention and intervention
services. In fact, of the students who do receive mental health services, 75% receive these
services in a school (Hoagwood & Erwin, 1997). Doll and Lyon (1998) proposed that
schools have opportunities to promote competence across academic, personal, and social
domains. Schools may prioritize these opportunities because they value skills that are
necessary for success in life such as good problem-solving, academic proficiency, and
social and emotional competence. Because they value these skills, schools may be
motivated to mobilize their resources to provide children with the structure and
curriculum necessary for success and, averting failure. Perhaps most evident, schools are
increasingly motivated to make sure students are successful since the enactment of the No
Child Left Behind Act of 2001, a federal law that calls for increased accountability for
student performance. Increasing demands on students’ academic performance as well as
increasing recognition that social and emotional competence in youth is vital to
adjustment into adulthood calls for action. As Greenberg, et al. (2003, p. 470) strongly
concede: “There is a national consensus on the need for 21st century schools to offer more
than academic instruction if one is to foster success in school and life for all children.”
10 Social and Emotional Factors Influence Academic Achievement
Educators are increasingly discovering that social and emotional factors greatly
influence learning and academic achievement (Collaborative for Academic, Social, and
Emotional Learning [CASEL], 2004). A 1997 analysis by Wang, Haertel, and Walberg
revealed that of the top eleven factors affecting learning, eight were social and emotional
factors, including: classroom management, parental support, student-teacher interactions,
social-behavioral attributes, school culture, classroom climate, motivational-affective
attributes, and peer group. Understandably, children with social-behavioral attributes
such as mental health and/or social-emotional competency problems may have
difficulties learning (Greenberg, et al., 2003). A child who is chronically tearful, irritable,
lethargic, and socially isolated will have a more difficult time paying attention in class,
being alert to opportunities to engage in and respond to instruction, and demonstrating
performance via class participation and written work. It is unlikely that after-school study
clubs, detention, time spent in conflict at home over unfinished homework, or even
grade-level retention will sufficiently or even appropriately allow students to “catch up”
or turn around. Elias (2001) captures this sentiment, emphasizing that “unless students
are given strategies to regulate their emotions and direct their energies toward learning, it
is unlikely that added instructional hours or days will eventuate in corresponding amounts
of academic learning” (p. 131).
Enhancing Social and Emotional Competence
Educators and mental health professionals generally believe that enhancing social
and emotional competence in children is valuable, as social and emotional learning
strategies have been found to positively influence social and emotional competence and
11 overall school success (Zins, Bloodworth, Weissberg, & Walberg, 2004). Many
schools have successfully integrated strategies and programs to promote social and
emotional learning into the regular curriculum (Greenberg, et al., 2003). Social and
emotional learning (SEL) is defined as “the process through which we learn to recognize
and manage emotions, care about others, make good decisions, behave ethically and
responsibly, develop positive relationships, and avoid negative behaviors“ (Elias, Zins,
Weissberg, et al., 1997, as cited in Zins, et al., 2003). SEL is founded on the premise that
school-based prevention and intervention instruction should not be limited to academics,
but should also include instruction in the area of social and emotional learning
(Greenberg, et al., 2003). Depending on their purpose, SEL programs can be
implemented at universal, secondary, and tertiary levels. These programs teach youth
how to recognize and manage emotions, solve every-day problems, take other’s
perspectives, set prosocial goals, and learn interpersonal skills to effectively deal with
developmentally appropriate tasks (Payton, et al., 2000). Learning these skills should
positively impact students’ social, emotional, and academic competence and ultimately
lead to future healthy and productive contributions to society.
Social and emotional skills are associated with students’ success in school, not
limited to general measures of social-emotional competency or academic performance,
but also including students’ attitudes, behavior, and overall school performance (Zins, et
al., 2004). Examples of these include student motivation to perform, feelings of
attachment to school, levels of engagement, attendance, study habits, and quantitative
measures of performance via daily grades and tests (Zins, et al.). Internalizing problems
such as feelings and behaviors associated with depression and anxiety can noticeably
12 hinder students’ abilities to engage in the tasks necessary to have an overall, successful
school experience. A successful experience may be gauged by an ability to focus on
coursework, manage internally distracting thoughts pertaining to acute and/or chronic
stressors, regulate feelings of anxiety associated with coursework demand and test-taking,
and maintain appropriate and meaningful interpersonal relationship with peers and adults.
Internalizing Problems Co-morbid with Externalizing Problems
Sometimes these internalizing problems co-occur with behaviors associated with
externalizing problems. Such externalizing problem behaviors can include impulsive and
high-risk behaviors, acting-out and oppositionality, verbal non-compliance, tardiness,
truancy, bullying, and generally disruptive behavior. Not only are these behaviors
difficult for teachers, administrators, and parents to manage, they usually interfere with
and hinder students’ social, emotional, and academic development. Angold, Costello,
and Erkanli (1999) discussed the co-occurrence, or comorbidity, between internalizing
and externalizing problems in their meta-analytic review. They define the co-occurrence
of symptoms across different diagnostic groups as heterotypic comorbidity. For example,
behaviors consistent with Oppositional Defiant Disorder typically include non-compliant
and acting-out symptoms as compared with behaviors consistent with Major Depression
which typically include symptoms associated with lethargy, cognitive rumination and
social withdrawal. Angold, at al. also discussed the co-occurrence of heterotypic co-
morbidity in time whereby symptoms from both diagnostic domains can co-exist
simultaneously and lead to decreased psychosocial adjustment. Certainly the issue of
concurrent, heterotypic comorbidity raises the stakes when considering comprehensive
13 mental health issues for youth and SEL programs used to address prevention and
intervention.
Integrating Social-Emotional Concepts Into Academic Curricula
Increasingly, social and emotional learning programs and concepts have been
integrated into academic curricula to enhance student learning (Elias, 2004; Zins, et al.,
2004). For example, the Social Decision Making and Social Problem Solving program
applies a social and emotional learning framework to social, emotional, and academic
learning contexts. Efforts to infuse students with these strategies have led to students’
automatic application of problem solving strategies as well as empirical evidence for
social and emotional learning effectiveness in school settings (Elias). Further evidence
for the link between the use of social and emotional learning and school success was
demonstrated in Wilson, et al.’s (2001) meta-analysis of 165 school-based prevention
programs. Programs that utilized social and emotional learning led to improved school
attendance, a logical and necessary prerequisite to school success. Other social and
emotional learning curriculums have demonstrated positive results pertaining to
children’s social, emotional, and academic achievement over time (The Resolving
Conflict Creatively Program, Brown, Roderick, Lantieri, & Aber, 2003) as well as
decreases in aggressive and disruptive behavior and improvements in the classroom
environment (PATHS Curriculum, Brown, Roderick, Lantieri, & Aber, 2003).
Not only are social and emotional learning programs being integrated into
academic curricula, but social and emotional developmental standards within a public
educational system are increasingly finding explicit support from state government. For
example, the state of New York recently passed the Children’s Mental Health Act of
14 2006 which specifically incorporates the concept of social and emotional learning into
public schools’ educational guidelines (The New York State Office of Mental Health).
Not only is the state required to develop a mental health plan, similar to Illinois’
legislation, but the state’s Department of Education will develop social and emotional
development standards which all students will be expected to meet. Although
implementation of efforts to achieve the standards are considered voluntary at this time, it
seems highly plausible that public schools will use social and emotional learning
programs to promote this area of childhood development in an effort to meet such
standards.
Standards for Social and Emotional Learning Programs
To be used appropriately and ultimately successfully, SEL programs must (a)
have a solid theoretical basis and (b) be research-based (Greenberg, et al., 2003). The No
Child Left Behind Act of 2001 pioneered standards pertaining to academic instruction and
prevention/intervention programs, mandating the use of theory-driven and research-based
methods across all instructional domains, including social and emotional learning. We
know that when students receive an empirically-supported intervention program, there is
a greater probability that they will experience success (Telzrow & Beebe, 2002). Zins, et
al. (2004) argue that teachers, given federal, state, and local expectations for student
achievement and demands for empirically-based programs, expect that the instructional
materials and methods they use reliably predict and lead to improved academic outcomes
in their students. When considering social and emotional learning curricula, teachers’
adaptation and use of such materials will inevitably improve when SEL curricula are
empirically proven to produce positive and reliable behavioral and academic outcomes
15 (Zins, et al.). Most importantly, students deserve the very best instructional practices,
whether academic or SEL, to enable them to reach their fullest potential. More evidence
is needed to determine the effectiveness of newly developed, promising SEL curricula as
a universal prevention and early intervention program with students in the general
education setting.
Primary Competencies Needed to Promote Social and Emotional Learning
In 2000, The Collaborative for Academic, Social, and Emotional Learning
(CASEL) developed a framework listing the primary competencies needed to promote
social and emotional development in youth (Payton, et al., 2000). Linked to this
framework, qualities of SEL programs were identified that had effectively promoted the
primary competencies inherent to social and emotional development (Payton, et al.). This
effort was launched in response to the growing number of SEL prevention and
intervention programs that schools were using to address students’ social, emotional, and
behavioral problems. CASEL’s review culminated in a set of guidelines by which
educators can evaluate potential or existing curricula on the merits of their link between
the competency areas required to promote social and emotional learning and the
characteristics of the packaged curricula. Among the characteristics of SEL programs
known to promote social and emotional learning, three are central to this proposed study:
(a) programs should provide teacher training as part of facilitating implementation, (b)
programs should provide ongoing technical assistance to teachers during implementation,
and (c) programs should use a method by which implementation of the program can be
monitored, using the data to improve subsequent implementation efforts. In a recent
review of the state of evidence-based program implementation, Fixsen, Naoom, Blase,
16 and colleagues (2005) concur that these components are fundamental to producing
meaningful outcomes for teachers and students.
Teacher Involvement in Social and Emotional Learning
Given the integral connection between the presence or absence of social and
emotional skills and social-emotional and academic competency, teachers must be
knowledgeable in this area in order to effectively manage and teach students in the
classroom (Zins, et al., 2004). Teachers frequently are asked to expertly implement a
variety of new curricula to accomplish this goal, but often do not receive adequate
training or support. Understandably, they may become frustrated when questions arise
and in the absence of support, must rely on their best judgment to solve the problem.
Difficulties with implementation may inadvertently turn a seemingly “easy-to-use”
curriculum into a challenging and cumbersome task. Training via staff development in-
servicing is frequently a means by which teachers receive training on new educational
programs (Joyce & Showers, 1988). However, this “train-and-hope” method (Stokes &
Baer, 1977) does not provide the comprehensive support teachers need to improve their
instructional skills and consequently affect students’ skills in areas targeted for
prevention and intervention (Ager & O’May, 2001; Fixsen, at al., 2005; Joyce &
Showers, 1988; Noell, et al., 2005).
Consultation and Program Implementation
Across the instructional and behavioral consultation literature, the use of a
consultant to provide support and feedback to staff during a program implementation
process has repeatedly been shown to be effective as a means of improving staff
adherence to program implementation protocols and ultimately, students’ skills (Fuchs &
17 Fuchs, 1996; Gersten, Chard, & Baker, 2000; Noell, Witt, Gilbertson, Ranier, &
Freeland, 1997; Noell, et al., 2000; Noell, et al.,1997). Historically, the organizational
behavior and management literature has consistently demonstrated evidence for
consultants utilizing performance feedback to improve employee performance in the
workplace (Alvero, Bucklin, & Austin, 2001; Balcazar, Hopkins, & Suarez, 1985). In an
educational setting, Coissart, Hall, and Hopkins (1973) demonstrated a functional
relationship between teacher behavior and experimenter feedback, providing evidence
that a performance feedback process can affect teacher behavior in order to positively
impact individual student behavior. In recent years, performance feedback has been used
to improve teachers’ adherence to classroom interventions and improve student
behavioral and academic performance (Mortenson & Witt, 1998; Noell, Duhon, Gatti, &
Connell, 2002; Noell, et al., 1997; Noell, et al., 2000, Noell, et al., 2005).
The Need for Evaluating the Use of Consultation Support and Social and Emotional
Learning Programming
Research in educational settings using performance feedback has predominantly
utilized single subject design evaluating the effectiveness of interventions implemented
between teacher and student dyads (Jones, et al., 1997; Mortenson & Witt, 1997; Noell,
et al., 1997; Noell, et al, 2000; Witt, et al., 1997). A recent study by Noell, et al. (2005)
also investigated the use of performance feedback with teacher-student dyads in a
classroom-based intervention using group design. A review of the performance feedback
literature did not reveal the use of performance feedback with teachers implementing a
social and emotional learning curriculum. However, a performance feedback process has
been used with programs related to social and emotional learning. Although performance
18 feedback does not appear to be universally defined, Noell, et al.’s (2005) description of
performance feedback seems to uniformly represent its process: “Performance feedback
consists of monitoring a behavior that is the focus of concern and providing feedback to
the individual regarding that behavior” (p. 88).
Group design research evaluations of school-based, health promotion programs
during 1985-1993 often utilized a process similarly described, but did not use the term
performance feedback in describing these processes (Botvin, Baker, Dusenbury, Tortu,
Botvin, 1990; Connell, Turner, & Mason, 1985; Errecart, et al., 1991; Ross, Luepker,
Nelson, Saavedra, & Hubbard, 1991; Smith, McCormick, Steckler, & McLeroy, 1993).
Nevertheless, results of these studies indicated the need for technical support using a
performance feedback process to facilitate quality implementation and beneficial results
of school-based health prevention and intervention programs (Payton, et al., 2000). Given
the evidence demonstrating the effectiveness of performance feedback across single
subject and group design on improving treatment implementation and student outcomes,
the use of performance feedback to promote the effective use of a promising social and
emotional learning curriculum used with general education students appears most
relevant and plausible.
Performance Feedback as a Means for Providing Consultation Support
The ultimate goal of using performance feedback is to improve the performance
of the individual implementing the intervention, by way of eliciting changes in behavior
related to intervention implementation (Balcazar, Hopkins, & Suarez, 1985). As applied
to an educational setting, changes in adult behavior (i.e., teachers improving
implementation practices of a curriculum with students) should subsequently positively
19 impact student outcomes via improved treatment integrity. Noell, et al.’s (2000) study
illustrates the process by which performance feedback typically orchestrates adult
behavior change. Teachers are presented with data regarding their performance, usually
including positive and corrective feedback, questions are answered and recommendations
are made by the consultant regarding improvements that can be made to improve
treatment integrity. In lieu of recommendations, consultants sometimes deliver corrective
feedback and obtain a commitment from the client/teacher to implement the treatment as
intended (Mortenson & Witt, 1998).
Incorporating Motivational Interviewing Into the Consultation Process
The communicative process by which performance feedback occurs has been
described as a “discussion” that is conducted between the clinician and teacher
(Mortenson & Witt, 1997) or most recently, an evolution of the problem-solving process
(Noell, et al., 2005), presumably aligning with the collaborative nature found in the
behavioral consultative or problem-solving approach to school-based consultative
procedures (Kratochwill, Elliot, & Callan-Stoiber, 1997). While the problem-solving
process includes a step where an intervention is selected, the communicative process by
which this intervention is agreed upon is often left to a description of “collaborative” or
such relationship is implied. Incorporation of the communicative process known as
motivational interviewing may provide a complementary and more detailed description of
the process by which interventions are chosen following performance feedback, while
highlighting the importance of minimizing adults’ resistance to change behaviors
interfering with treatment integrity.
20 Motivational interviewing has been used in medical and public health settings
as a method to facilitate behavior change in clients who struggle with behaviors
interfering with a variety of areas of functioning and where change is often met with
resistance (Miller & Rollnick, 2002). Motivational interviewing has most notably been
used with individuals having alcohol drinking problems (Brown & Miller, 1993). The
goal of motivational interviewing is to help a person move toward changing behaviors to
enhance their physical, mental, and behavioral health via a communicative process that is
collaborative and supportive. The clinician works with a client by procuring information
rather than instructing the client as to what should or must be done, supporting the client
instead of persuading, and recognizing the client as the only one who can bring about
behavioral change that promotes good health. As part of the motivational interviewing
process, a “menu of options” is frequently generated between the clinician and client in
an effort to direct a course of action that is client-driven and clinician-supported.
Motivational interviewing in conjunction with performance feedback in a
classroom setting was examined by Reinke (2005) in an effort to improve teacher
performance. A Classroom Check Up procedure was modeled after the Drinker’s Check
Up (Miller, Sovereign, & Krege, 1988). In Reinke’s study, classroom performance data
were collected which the consultant reviewed with the teacher. Following this review, the
consultant and teacher generated a menu of options in which potential areas needing
improvement were discussed by the teacher and consultant, and an agreement was
reached as to the area to be focused on for improvement. Combining a component of
motivational interviewing (i.e., generating a menu of options in a manner that is client-
driven and clinician-supportive) with performance feedback provided a discriminative
21 addition to the communication process by which performance feedback is delivered.
Furthermore, an effort to use a standardized approach to performance feedback via an
empirically-based method of addressing resistance to behavioral change (i.e.,
motivational interviewing) serves to address concerns by Sheridan, Welch, and Orme
(1996) that the process by which behavioral consultation services are delivered must
describe in more detail the procedures used during the consultation process, rather than a
mere statement that these services were provided. This issue is also true in the
performance feedback literature whereby the process in which feedback is delivered is
either directed by the consultant, devoid of consideration of resistance-to-change issues
on the part of the client/teacher.
The Value of Measuring Treatment Integrity
Performance feedback incorporates a measure of treatment integrity as a primary
means of delivering feedback to consultees that can facilitate positive student outcomes
(Noell, Gresham, & Gansle, 2002). Measuring and monitoring treatment integrity, or the
extent to which a program is implemented as intended, is vital when establishing the
effectiveness of a program on a targeted outcome variable (Gresham, 1989; Telrow &
Beebe, 2002), such as reducing internalizing symptoms and increasing students’
knowledge of coping strategies. Not only is a measurement of treatment integrity
necessary to strengthen the internal and external validity of the study (Moncher & Prinz,
1991), measuring treatment integrity has proven invaluable in the performance feedback
literature when linking degrees of teacher implementation to levels of student success
(Witt, et al., 1997). Likewise, measures of treatment integrity have linked social and
emotional learning-type community interventions to consumer outcomes (e.g.,
22 Multisystemic Therapy: Henggeler, Melton, Brondino, Scherer, & Hanley, 1997; social
skills training: McEvoy, Shores, Wehby, Johnson, & Fox, 1990; and treatment for
anxiety disorders, Vermilyea, Barlow, & O’Brien, 1984).
Observation sessions are required to measure the rate of treatment integrity and in
turn, provide performance feedback. A study by Witt, et al., (1997) demonstrated a large
effect on student performance when feedback was delivered daily (i.e., 100% of the
intervention sessions the performance feedback was delivered). However, it is often not
necessary nor practical to observe all treatment sessions and provide performance
feedback for every session that is delivered. For example, in a study where a classroom
intervention was implemented on a daily basis, Mortenson and Witt (1998) used a
performance feedback procedure at a rate of one observation and performance feedback
session for every five instances the intervention was delivered (or 20%). Significant
change was measured in teacher performance using this ratio, suggesting there may be a
standard for providing performance feedback that is not only effective, but efficient.
Although measuring treatment integrity in this study served as a tool via
performance feedback to answer a primary research question of whether consultative
support in the form of performance feedback leads to better student outcomes using a
promising SEL program, its value cannot be underestimated. Unfortunately, the absence
of adequate levels of treatment integrity when implementing programs in community
settings may lead to less effective outcomes as compared to outcomes in research settings
where treatment integrity is more often monitored (Weisz, Melton, Brondino, Scherer, &
Hanley, 1997). In educational settings, the absence of measuring treatment integrity (as is
often the case) may lead to deviations from the prescribed components and/or
23 implementation methods required by an evidenced-based curriculum. Deviations may
be due to teachers’ misunderstanding of program implementation procedures, time
limitations, classroom management problems, instructional skills deficits, and other
challenges and barriers. Given these challenges, perhaps the primary motivator to use
performance feedback as a means to improve outcomes is best captured by Hengeller and
colleagues’ (1997) decisive opinion that the effort and cost of providing community
settings with cost-efficient training and consistent monitoring of program implementation
with the intent to at least approximate research setting-like results far outweigh the cost
of providing inadequate services (i.e., implementing a program without maximizing its
potential with the use of validated treatment integrity monitoring methods).
Used with a promising SEL program, a consultant who provides regular
performance feedback during program implementation meets CASEL’s recommended
guidelines for the implementation of quality SEL programs (i.e., provide teacher training,
provide ongoing technical assistance to teachers during implementation, and use data to
improve implementation) as well as the most recent review of research on the
implementation of evidence-based programs (Fixsen, et al., 2005). Using these
procedures with an SEL program should lead to meaningful and positive outcomes for
youth.
The Strong Kids Social and Emotional Learning Curriculum
A social and emotional learning program that has demonstrated increasing
promise is the Strong Kids program a 12 lesson, skill-based SEL curriculum, developed
by members of the Oregon Resiliency Project Research Team at the University of Oregon
(Merrell, Carrizales, et al., 2006). It is based on the premise that resiliency, or the ability
24 to cope during adversity, can be learned when core skill areas are taught. Strong Kids
is designed to be used with students in grades 4-8 in prevention and intervention efforts
in regular education classrooms and other educational settings. Its design is based on
evidence-based instructional design principles, such as the use of scripts, activating prior
knowledge, and providing opportunities for practice. In addition, Cowen’s (1994) “five
pathways to psychological wellness” are incorporated as a core concept within the
curriculum. These pathways include: (a) early attachments that are “wholesome”, (b)
becoming competent with developmentally appropriate skills, (c) being exposed to
settings that encourage wellness, (d) feeling a sense of empowerment or being in control
of one’s future, (e) possessing coping skills to deal with stress effectively. The lessons
focus on building social and emotional skills and strategies, primarily addressing
internalizing symptoms such as behaviors, thoughts, and feelings associated with
depression and anxiety. Strong Kids was designed to be easy to implement, strength-
based and positive, work within an existing curriculum or school setting, require minimal
training, time, and financial resources, use structured but flexible scripts, and provide
frequent opportunities for practice.
The Need to Study Strong Kids With a General Education Population
Prior pilot research has tentatively established the effectiveness of Strong Kids
with children in general education settings (Merrell, Juskelis, Buchanan, & Tran, 2006).
Children who participated in the curriculum demonstrated a powerful increasing trend in
their knowledge of skills to cope with difficulties. Data collected to date have not
consistently demonstrated a desired decrease in internalizing emotional problem
symptoms. This finding could be due to a variety of factors including, but not limited to,
25 the sensitivity of the outcome measures used and a ceiling effect attributed to the
population studied (i.e., general education children may report few internalizing
symptoms at pre-test therefore leaving little room for improvement in symptoms). There
is a need to further the study the effectiveness of Strong Kids in the general education
classroom to determine whether children continue to gain knowledge of coping skills and
whether it can positively impact existing internalizing symptoms for students at-risk of
developing associated difficulties.
An Interest in Evaluating Strong Kids in Conjunction With Consultation Support
In addition to the continued need for additional evidence that Strong Kids is an
evidence-based SEL program for middle school students in the general education setting,
there are no studies that have examined the effect that regular on-site consultation with
teachers using performance feedback has on have on students’ social and emotional
competency. Although it is strongly recommended that teachers be given regular support
when implementing an SEL program (Payton, et al., 2000) and performance feedback has
proven to be an effective means of improving student outcomes in the classroom (Noell,
et al., 2005; Witt, et al., 1997), it had not been established whether giving teachers this
support will lead to improved student outcomes when using Strong Kids. When teachers
are given support and feedback during the implementation process, they are more likely
to have an easier time implementing the program, the program will be delivered more
closely to its prescribed method because implementation will be more closely monitored,
and teachers will receive relevant feedback to improve implementation skills. Ultimately,
when Strong Kids is implemented more closely to its intended method, students should
experience optimal outcomes.
26 The Importance of Evaluating Teachers’ Acceptability of Strong Kids
A final contribution this project intended to make was the examination of
teachers’ beliefs regarding the effectiveness, feasibility, and acceptability of the Strong
Kids curriculum. Telzrow and Beebe (2002) indicated that the extent to which classroom
teachers consistently and correctly implement a program depends on the belief that
programs are effective at producing positive change in students and are relatively easy to
implement. Schwartz and Baer (1991) defined this concept, social validity, to be the
extent to which an intervention is deemed acceptable, relevant, and viable on behalf of
the individual(s) implementing an intervention. Social validity is paramount to the
success of a new SEL curriculum because the likelihood of the program being
implemented as designed hinges on whether or not the user believes it to be meaningful
and have realistic utility in natural settings. In order for new curricula to have at least
adequate social validity, research must take into account what the consumer deems
socially important.
Strong Kids was designed to be a time and cost efficient program, logically and
naturally embedded into a typical school day schedule. As teachers have increasing
demands on their time and expectations to meet high accountability standards, their buy-
in and confidence in an SEL program’s ability to be “time and money well spent” are
critical if evidence-based SEL programming has a future of being implemented school-
wide. Furthermore, recent evidence suggests teachers who received performance
feedback perceived the process as supportive and positive (Noell, Guhon, Gatti, &
Connell, 2002). An inevitable outgrowth of teacher confidence is their willingness to
increasingly expose students to SEL programs, such as Strong Kids, to prevent social and
27 emotional problems from occurring or escalating and impacting their school
performance. Because social and emotional skills have been linked to student
performance (Greenberg, et al., 2003), and later life adjustment, it is vital that SEL
programming become a part of the general education curriculum.
To measure consumer acceptability of a program, Wolf (1978) offered three
guidelines when structuring an assessment method. He suggested asking whether the
goals of the program are designed to meet the same goals of the intervention
implementers, whether the procedures leading to outcome goals are acceptable to the
participants, and whether the participants are satisfied with the outcomes of the
intervention. These (1978) guidelines were used to determine the level of acceptability
Strong Kids has with teachers who implement it in the general education setting.
Given the increasing interest and promise in using social-emotional learning
programs in school settings, it seems timely, appropriate, and useful to continue to
evaluate the effectiveness of a new social-emotional learning program in a general
education classroom setting. Exploratory investigation of the use of consultation support
when implementing a social-emotional learning program should provide introductory
information regarding the usefulness of this technology in this context. Finally,
information pertaining to teachers’ attitudes toward social-emotional learning in general
and, the Strong Kids curriculum in particular, will provide invaluable consumer feedback
that is useful for program refinement.
28
CHAPTER III
METHOD
Design
A randomized control group pretest-posttest experimental design was used for this
study (Isaac & Michael, 1995). Student participants were nested in Language Arts Block
classes and although ostensibly assigned to these classes in a somewhat random manner,
it could be argued that the method of nesting students within classrooms was more
“quasi” experimental than a true random assignment procedure. The two independent
variables for this study were: 1. Time, with two levels: pretest and posttest; and 2. Group,
with three levels: Group 1 (Control Group), Group 2 (No Consultation), and Group 3
(Consultation). Group 1 did not receive the curriculum and Groups 2 and 3 received the
Strong Kids curriculum. Table 1 displays the design and sample size used for this study.
Table 1
Research Design
Time 1: Pretest Time 2: Posttest
Group 1: Control (n = 46) X X
Group 2: Strong Kids with No
Consultation (n = 40)
X X
Group 3: Strong Kids with
Consultation (n = 39)
X X
29 Participants and Setting
Participants were selected from existing sixth grade classroom configurations at
Thurston Middle School in Springfield, Oregon. Springfield is located in the Central
Willamette Valley and has a population of approximately 54,000 residents. Thurston
Middle School had 547 students enrolled at the time of this study. 90% of these students
were listed in school records as White, 2% as Black, 3% as Hispanic, 3% as American
Indian, 1% as Asian, and 1% were listed as Unspecified. In terms of socioeconomic and
special education status of the school population, 33.30% of the total student body
qualified for the free and reduced lunch program, and 14% of the students were receiving
special education services. The free/reduced lunch and special education data for this
school are similar to the U. S. averages for these categories. The Strong Kids curriculum
was taught in the general education classroom during a 50 minute time period during
which language arts is typically taught.
Student participants. Student participants included three, 6th grade, regular
education, language arts teachers who were selected based on their willingness to
participate in this project, and 125 general education students. Students who were
identified as requiring special education instruction in the area of reading did not
participate in this curriculum. School personnel elected to adopt the Strong Kids
curriculum, therefore all students who received general education language arts
instruction were eligible to receive the curriculum. Out of 139 students who were
eligible, 10% of these students (n = 14) either elected to not participate in the study, their
parents indicated they did not want their child to participate, or the student had
incomplete data due to absence or withdrawal from the school at either pretest or posttest.
30 Of the 125 students who participated, 55% were male and 45% were female. Of these,
52% were 11 years old and 48% were 12 years old at the time of this study. Table 2
indicates demographic information for the 6th grade students who participated in this
study.
Table 2
Demographic Information of Student Participants
Gender n in Group 1 n in Group 2 n in Group 3 Total N
Male 25 22 24 71
Female 21 18 18 57
Total 46 40 42 128
Age
11 25 26 16 67
12 21 14 26 61
Total 46 40 42 128
Teacher participants. Teacher participants were three, 6th grade, general
education, language arts teachers at Thurston Middle School who were selected based on
their willingness to participate in this project. Each of the three teachers taught two
language arts classes per day, each class having an average of 21 students per class. One
of the teachers was on paternity leave at the time of recruitment and opted to not teach
the curriculum when he returned. Subsequently, this teacher was assigned to the control
group and the other two teachers were randomly assigned to treatment groups. The
teacher for Group 1 was male and the teachers for Groups 2 and 3 were female.
31 Consultant. The consultant was the student researcher, a school psychology
doctoral candidate and licensed school psychologist with nine years experience working
as such with teachers in the general education setting. The consultant was well-versed in
the evidence-based conceptual framework of the Strong Kids curriculum as well as all
instructional components, having recently participated in revising all lessons, and having
previously been involved over the 2005-2005 school year consulting with an educational
specialist on the implementation of Strong Kids with 4th and 5th grade students.
Independent Variables
This study had two qualitative independent variables, Time and Group. Time had
two levels: Time 1 (Pretest) and Time 2 (Post-test). Group had three levels: Group 1
(Control Group), Group 2 (No Consultation), and Group 3 (Consultation).
Time. At Time 1, the quantitative pre-test dependent measures were administered
to student participants during Lesson One of the Strong Kids curriculum. The same
dependent measures were administered 13 weeks later and at Time 2. Additionally, a
qualitative posttest dependent measure was given to teacher participants to evaluate their
attitudes toward the curriculum.
Group. Students in Group 1 did not receive the curriculum. Students in Groups 2
and 3 received the Strong Kids curriculum. Group 1 was comprised of the students of the
teacher who did not implement the curriculum. Group 2 was comprised of the students of
the teacher who implemented the curriculum and received minimal consultation. Group 3
was comprised of the students of the teacher who also implemented the curriculum and
received consultation in the form of performance feedback and weekly phone or
electronic mail support. Table 3 summarizes the groups, their treatment group status, the
32 total number of participants in each group, whether students received the curriculum,
and whether the participants completed the dependent measures.
Table 3
Summary of Group Status
Group
Number
Treatment Group N Received Curriculum? Received Dependent
Measures?
Group 1 Control 46 No Yes
Group 2 No Consultation 40 Yes Yes
Group 3 Consultation 39 Yes Yes
Dependent Variables and Measures
This study had three quantitative dependent variables and one qualitative
dependent variable. Dependent variables that were measured include: student knowledge
of social and emotional coping strategies, student-reported symptoms of behaviors and
feelings associated with depression and anxiety, a general indicator of externalizing
problems, and teacher acceptability and satisfaction with the Strong Kids curriculum.
Quantitative dependent measures. The first dependent variable, student
knowledge of social and emotional coping strategies, was measured using a 20-item, self-
report questionnaire comprised of true-false and multiple-choice questions (See
Appendix A: Strong Kids Knowledge Test). This questionnaire was developed
specifically for the Strong Kids curriculum and has been used in prior pilot research.
Questions are based on concepts found throughout the 12 lessons of the curriculum. The
purpose of the questionnaire is to determine the extent to which students have learned key
33 social and emotional coping strategies as a result of being exposed to the Strong Kids
curriculum. Previous pilot testing of this 20-item measure has shown that it is sensitive to
student changes in knowledge as a result of participating in the program, and that is has
an internal consistency reliability (Cronbach’s alpha) in the .60’s to .70s range, which is
considered adequate for a research measure of this length that is not used for making
decisions regarding individual students.
The second quantitative dependent variable, students’ symptoms of behaviors and
feelings associated with depression and anxiety, was measured by two self-report
questionnaires. The first questionnaire, the Children’s Depression Inventory (See
Appendix B), is a 27-item questionnaire which measures children’s negative mood,
interpersonal problems, feelings of ineffectiveness and anhedonia, and negative self-
esteem (Kovacs, 1991). Respondents chose among three response choices as to the item
that best describes him/herself in the past two weeks. This questionnaire requires reading
skills at minimum of a first grade level. Internal consistency has generally been found to
be in the .80s and test-retest reliability between .70-.85 for short and longer (several
months) time intervals. The CDI has been found to be highly correlated with other
instruments that measure internalizing symptoms and is a measure that has been widely
researched for use as a self-report instrument with children (Merrell, 2003).
Students’ symptoms of behaviors and feelings associated with depression and
anxiety was also measured using the Internalizing Symptoms Scale for Children (ISSC)
(Merrell & Walters, 1998) (See Appendix C). The ISSC is a 48-item self-report
assessment for students in grades 3-6, designed to measure the cognitive, affective, and
behavioral characteristics associated with depression and anxiety. The ISSC includes an
34 empirically-derived subscale structure that includes two separate factors (positive
affect, and negative affect/general distress), as well as a total symptoms score. Several
previous studies, as well as information in the ISSC manual, have shown the measure to
have very high internal consistency reliability (.90 and higher), strong test-retest
reliability at short to medium time intervals (.70 to .80), strong convergent validity with
other self-report measures (including the Children’s Depression Inventory, Youth Self-
Report, Revised Children’s Manifest Anxiety Scale, and Behavioral Assessment System
for Children (Merrell, Blade, Lund, & Kempf, 2003). The ISSC has been shown to
accurately differentiate between children referred to clinics for internalizing symptoms,
children referred to clinics for non-internalizing symptoms, and non-referred children and
has significantly correlated with broadband scores on the Child Behavior Checklist
(Merrell, McClun, Kempf, & Lund, 2002).
The third quantitative variable was a measure of office disciplinary referrals. The
purpose of measuring office disciplinary referrals for this study was to measure a general
indicator of externalizing behaviors, as externalizing behaviors have been found to co-
occur with internalizing behaviors (Angold, et al., 1999). The total number of referrals
for each student participant was obtained from school records at pre- and post-test, as
well as the number of minor and major infractions that comprised this total score. Major
office disciplinary referrals were behaviors described as non-compliant, bullying, and
disruptive. Minor office disciplinary referrals were rule infractions that included tardy
and minor disruptive behavior and inappropriate language.
Qualitative dependent measure. Teachers’ attitudes toward the Strong Kids
curriculum was assessed via a brief, 32-item questionnaire (Strong Kids Social Validity
35 Survey) completed by teachers in Groups 2 and 3. Teachers were asked questions
across five domains: the alignment of goals between the teachers and the curriculum, the
acceptability of the procedures used implement the curriculum, satisfaction with observed
results of using the curriculum, the feasibility of implementing the curriculum, the
importance of implementing the curriculum, and teachers’ confidence in implementing
the curriculum, and finally, open ended questions regarding general likes and dislikes
toward the curriculum. The questions in the interview and survey are based on Wolf’s
(1978) guidelines for assessing social validity and will focus on the following areas: (a)
the extent to which the goals of the curriculum (i.e., to increase students’ knowledge of
social and emotional coping strategies) the same as those of the teachers’; (b) the extent
to which the procedures of the implementation of Strong Kids are acceptable to the
teachers; and (c) the extent to which the teachers are satisfied with student outcomes.
Teachers were asked to respond using a forced-choice respond method to choose among
three response choices. For example, for the question: “How important is it that students
experience fewer social, emotional, and behavioral problems?” the respondent was asked
to choose one of the following: very important, somewhat important, not important.
Teachers were then asked to write responses to three open-ended questions.
Scoring procedures for dependent measures. Students’ responses on the
quantitative measures were entered, scored, and tabulated using the SPSS 11.0 for Mac
program, and both item-level and summative total data were calculated. The researcher
served as the primary scorer of all dependent measures at Time 1 and Time 2. Teacher
responses on the study’s qualitative measure were reviewed and themes were
consolidated.
36 Treatment integrity measure. Treatment integrity was measured to: 1)
determine the level of integrity that teachers for Groups 2 and 3 achieved when
implementing a lesson and 2) as a means to collect treatment integrity data that were used
for the performance feedback component of consultation for the teacher in Group 3.
Treatment integrity was measured for seven of the 12 lessons (e.g., Lessons 1, 2, 3, 4, 5,
8, and 11) in Strong Kids via direct observation of teachers for Groups 2 and 3. The
student researcher observed and collected integrity data for the teacher in Group 3;
another trained observer observed and collected integrity data for the teacher in Group 2.
Treatment integrity data were collected during the entire duration of lesson
implementation, approximately 50 minutes. A checklist that reflected the primary
components of each lesson was used for data collection (See Appendix E). The observer
endorsed each component if it was observed to have been implemented. The total number
of components that were implemented was summed and divided by the total number of
components that should have been implemented, the product multiplied by 100% to yield
the percentage of components implemented for each lesson observed. For the teacher
who received consultation, the percentage of components implemented was graphed on a
data sheet and used to provide performance feedback. (See Appendix F). Teachers in
Groups 2 and 3 were not told when treatment integrity observations would be conducted
to avoid any preparation that a teacher might do that could potentially inflate integrity
measurements. Treatment integrity data indicated both teachers implemented the
components for each lesson at a high rate. On average the teacher for Group 2
implemented 95% of the components in each lesson and the teacher for Group 3
37 implemented 87% of the components in each lesson. The results of treatment integrity
data collected for Groups 2 and 3 are summarized in Figure 1.
Note. Percentage indicates the ratio of components implemented out of total possible for each lesson. Figure 1. Summary of Treatment Integrity Data over Time
Inter-observer reliability for treatment integrity measure. Inter-observer
reliability was assessed for four of the seven (57%) of the lessons where treatment
integrity data were collected via in-class observation. Two graduate students
knowledgeable about the Strong Kids curriculum served as treatment integrity inter-
observer reliability data collectors. For each lesson, the number of components that both
observers (e.g., the student researcher and inter-observer reliability data collector) agreed
were implemented was summed and divided by the total number of components in the
lesson, the product then multiplied by 100%. Average inter-observer agreement between
38 the data collectors assigned to Group 2 was 94%. Average inter-observer agreement
between the data collectors assigned to Group 3 was 100%.
Procedures
Recruitment. Emails were sent to Springfield School District’s research
compliance officer as well as the principal of Thurston Middle School (see Appendix G).
The purpose of this email was to recruit teacher and student participants for this study
and to ascertain interest in this study. Verbal agreements were obtained from these
administrators. Interested teachers were sent an information handout via email to provide
additional information regarding the study (see Appendix H).
Consent procedures. Two weeks prior to the start of data collection, parents of 6th
grade, general education students at Thurston Middle School were sent via U.S. Mail
passive consent letters (See Appendices I and J), that had been approved by the school
principal, the district’s research liaison, and the University of Oregon Human Subjects
IRB. The school’s administrative secretary was given a copy of this letter, a copy of the
curriculum for parent reference, and was asked to document correspondence from parents
who did not want their child to participate in the study. Parents who called or wrote
requesting either that their child not participate in this study or who called with questions
were called by the researcher. The names of students that were indicated by parents as not
participating in the study were recorded so that the students would be excluded from data
collection procedures. Students in Groups 1, 2, and 3 assented to participating to the
study on the day in which Groups 2 and 3 began their participation in the Strong Kids
curriculum (see Appendices K, L, M). Those students who chose to not participate in the
dependent measure data collection were excused to the library for a supervised study
39 time. Teachers consented to their participation in this study during their training
session (see Appendix N).
Training. A week prior to the implementation of the curriculum, teachers in
Groups 2 and 3 received a two-hour training where they received information regarding
the theoretical basis for the program and the curriculum materials. Participants were then
given an overview of the 12 lessons in the curriculum. Discussed was the organization of
the content of the lessons, a description of the materials that may be needed during the
course of implementation, and discussing portions of lessons that may be more difficult
to implement. Any questions the teachers had regarding the curriculum and
implementation were answered at this time. Prior studies have shown that a review of and
instruction on training materials is not sufficient for reliable implementation of program
procedures (Harchik, Sherman, Hopkins, Strouse, & Sheldon, 1989; Noell, et al., 1997).
Thus, it is not anticipated that a two-hour introductory training confounded the
independent variables Time and Group. The teacher for Group 1 then joined the teachers
from Groups 2 and 3 for a 30 minute training session regarding data collection
procedures (see Appendix O for training agenda). Teachers for Groups 1, 2, and 3 then
signed consent forms to participate in this study. Finally, the teachers for Groups 2 and 3
were randomly assigned to treatment conditions and dates were set to begin data
collection and program implementation.
Group 1: No treatment control. Students in Group 1 did not receive the Strong
Kids curriculum, but served as a control group to the two treatment groups. The teacher
for this group allowed two, 30 minute blocks of time during language arts instruction
time in which students completed the dependent measures at pretest and Post-test. These
40 students completed the dependent measures on the same day in which students in the
treatment groups completed the dependent measures. Thurston Middle School has
strongly considered adopting this curriculum for long-term use for all grade levels. It is
believed that this group of students will receive the curriculum during their seventh grade
year.
Groups 2 and 3: Treatment with and without additional consultation.
Implementation of Strong Kids curriculum. Teachers for Groups 2 and 3 implemented
Strong Kids at a rate of one lesson per week, with the exception of a two-week span
between lessons when students were not in school due to Spring Break recess. Lessons
were implemented during the 50-minute block of time used to teach language arts. A
Strong Kids lesson was taught in place of the usual language arts lesson. Dependent
measures were administered prior to beginning Lesson 1 and following the completion of
Lesson 12.
Group 2: No performance feedback condition. One week prior to the
implementation of Lesson 1, the teacher for Group 2 was given a “Frequently Asked
Questions” handout (see Appendix P). This handout listed common questions regarding
curriculum implementation and listed the contact information for this researcher should
an emergency arise. The researcher indicated to this teacher that except for an emergent
situation, the student researcher would not be in contact with this teacher during the
duration of curriculum implementation. This handout was provided in order to maintain a
supportive and positive relationship between the university and educational setting and
provide a minimum amount of support to ensure ethical prudence. No contact was made
41 with this teacher during the implementation of the curriculum, with the exception of
arranging for a pizza party for the last day of curriculum implementation.
Group 3: Performance feedback condition. The teacher for Group 3 received
consultation via intermittent performance feedback throughout the duration of the
implementation of Strong Kids. After treatment integrity data were collected, the results
were graphed on a master graph. Balcazar, et al., (1986) suggested the use of visual,
graphic representation to show a consultee the results of treatment integrity data
collection and, several studies utilizing performance feedback have applied this method
(Mortenson & Witt, 1998; Noell, et al., 1997, Noell, et al., 2000, Noell, et al., 2005).
Although a study by Witt, et al., (1997) demonstrated a larger effect on student
performance when feedback was delivered daily (i.e., 100% of the intervention sessions
the performance feedback was delivered) as compared with Mortenson & Witt’s results
with weekly feedback, this study used performance feedback at a rate of 58% of lessons
taught delivered at an interval where feedback was concentrated in first several lessons,
then spread to every other lesson. Given prior studies’ methods and results, performance
feedback delivered at a rate of 58% of lessons implemented was considered reasonable to
test the hypothesis that performance feedback would lead to improved student outcomes.
This procedure was believed to represent a realistic amount of consultation time that
school psychologists or other school-based consultants are able to provide in a natural
setting.
During the teacher’s preparation time, approximately one hour following integrity
data collection, the researcher met with the teacher for approximately 10 minutes to share
graphed data and deliver performance feedback. Feedback was concentrated in the first
42 several lessons, then spread to every other lesson. The consultant reported the
percentage of components implemented and reviewed the specific components
implemented or not implemented. The master graph was shared with the teacher at each
time performance feedback was delivered to illustrate the teacher’s integrity data over
time.
Performance feedback incorporated the following components: (a) graphed
treatment integrity data were shared with the teacher, (b) a minimum of three praise
statements were delivered for areas noted in which the teacher implemented the
components and/or performed notably well, (c) a menu of options was collaboratively
generated regarding areas in which improvements could be made; these options were
initially generated by the researcher based on the areas in which integrity was not
achieved; however, teacher input was consistently solicited in terms of these or other
areas in which improvements were believed to be made and the final decision for the
identified area of improvement was based on the teacher’s priorities and wishes, (d)
resources that the teacher might need to implement an identified are of improvement
were identified and discussed with the teacher, and (e) the area for improvement was
decided upon by the teacher and reported to the researcher. A checklist was used to
certify that all components of the performance feedback process were implemented. (See
Appendix Q). Responses to the items on the checklist were dichotomous, indicating that
the component was either implemented or not implemented.
In addition to seven performance feedback sessions, the consultant initiated a
brief check-in, either via phone or electronic mail, with the teacher following the
implementation of the lessons when treatment integrity data and subsequent performance
43 feedback sessions were not conducted in the classroom. The following issues were
addressed during this check-in session: (a) whether the teacher had any questions
regarding implementation issues, (b) a discussion of ways in which they could be
addressed, and (c) general appreciation statements regarding the teacher’s efforts. The
consultant used a checklist to certify all components of the brief check-in sessions were
implemented (See Appendix R). Responses to checklist items used a dichotomous
measure indicating that the component was implemented or it was not implemented.
Integrity of performance feedback procedures. Integrity of performance feedback
procedures was self-reported by the consultant providing feedback. Integrity was
measured at an average rate of 100% of lessons where feedback was given. Inter-observer
reliability was assessed for four out of the 7 lessons (57%) where performance feedback
was delivered. The inter-observer reliability data collector observed the consultant and
the teacher during the 10 to 15 minute performance feedback session. The observer used
the same checklist that the consultant used to certify that all components of performance
feedback were implemented. The number of components of the performance feedback
session in which the consultant and observer indicated the same components were
implemented were summed and divided by the total number of components that should
have been implemented, the product then multiplied by 100%. Inter-observer agreement
was measured to be 100%.
Qualitative dependent measure administration. Following the implementation of
the administration of the quantitative dependent measures at Time 2, teachers in Groups 2
and 3 completed the Strong Kids Social Validity Survey. Surveys were reviewed and
responses were grouped into themes across the two respondents.
44
CHAPTER IV
RESULTS
This chapter includes a description of the analyses used to evaluate the data for
this study and the results of these analyses. Results are reported in order of research
questions proposed. In addition to the results pertaining to these questions, analyses were
conducted to evaluate internal consistency for dependent measures as well as convergent
and discriminant validity among these measures.
Analyses
Two-way, repeated measures analyses of variances (ANOVAs) were conducted to
examine the effects of the Strong Kids curriculum across a 13-week time period for the
three study groups. The ANOVAs were conducted for all quantitative dependent
measures for students’ knowledge of social-emotional coping skills and specific
curriculum content, students’ self-report of internalizing symptoms including those
associated with negative and positive affect, and office disciplinary referrals as reported
by the school. Descriptive statistics for these measures are reported in Table 4 and Table
11. Teachers’ attitudes toward social-emotional learning and the Strong Kids curriculum
were measured with a brief qualitative survey measure.
45
Table 4
Means and Standard Deviations for Self-Report, Quantitative Dependent Measures, Listed for Pretest and Posttest Assessments
Measure Control No Consultation Consultation
Mean SD Mean SD Mean SD
Knowledge
Pretest 12.14 3.51 12.73 3.45 14.22 3.32
Posttest 12.36 3.11 15.60 2.99 16.15 2.64
ISSC
Pretest 45.04 19.26 44.85 26.96 45.80 23.39
Posttest 45.11 20.64 39.73 24.75 45.24 22.34
ISSC Factor 1
Pretest 38.26 16.32 37.60 20.97 38.59 19.66
Posttest 38.28 16.41 33.80 18.67 37.69 18.63
ISSC Factor 2
Pretest 8.85 6.86 9.85 9.74 9.74 9.14
Posttest 9.33 8.12 8.05 9.66 8.77 7.65
CDI
Pretest 5.87 5.67 6.82 9.41 6.00 7.41
Posttest 7.36 9.05 6.50 9.36 6.00 5.89
46 Effect of Strong Kids on Knowledge of Healthy Social-Emotional Behavior
To answer the first research question: What is the effect of participation in the
Strong Kids social and emotional learning curriculum on knowledge of healthy social-
emotional behavior among middle school students in a general education setting?, a two-
way repeated measures ANOVA was conducted to evaluate the effect of the Strong Kids
curriculum, implemented over the course of 13 weeks, on knowledge of social-emotional
coping skills. The dependent, repeated measures variable was the Strong Kids Knowledge
Test. The within-subjects factor was Time (pretest and posttest), and the between-subjects
factor was Group (Control, No Consultation, and Consultation). The Time by Group
interaction effect and Time and Group main effects were tested using the multivariate
criterion of Wilks’ Lambda (Λ). The ANOVA revealed a significant interaction between
Group and Time, Λ = .85, F(1, 122) = 10.89, p < .01, indicating a difference in
knowledge of social-emotional skills between groups and across time. Observed power
using Wilks’ Lambda was .99, p = .05, suggesting a very low chance of making a Type II
error; that is, failing to reject a false null hypothesis. Table 5 shows the results of the
ANOVA.
47 Table 5
Mixed 2-Way ANOVA for the Interaction Effect of Group and Time on Knowledge of Social-Emotional Skills as Measured by Strong Kids Knowledge Test
Source df F η p
Between subjects
Group 2 12.25* .17 <.01
Error Between 122
Within subjects
Time 1 50.17* .29 <.01
Group 2 12.25* .17 <.01
Time * Group 2 10.89* .15 <.01
Error Within 122
* p < .05.
Post-hoc pairwise analyses using the Bonferroni method to control for Type I
error indicated the differences in mean raw scores on the knowledge test were significant
between the control group and the no consultation group (-1.91, p = .01) and between the
control group and the consultation group (-2.93, p = <.01). Figure 2 illustrates these
significant differences, indicating that for the control group, the mean raw score on the
knowledge test remained the same from pretest to posttest. The mean raw knowledge test
score for the no consultation group and the consultation group increased from pretest to
posttest, and were both significantly different than the mean raw knowledge test score for
the control group.
48
Figure 2. Knowledge of Social-Emotional Skills Effect of the Strong Kids Curriculum on Internalizing Symptoms
To answer the second research question: What is the effect of participation in the
Strong Kids social and emotional learning curriculum on internalizing symptoms
associated with emotional-behavioral problems among middle school students in a
general education setting?, a two-way, repeated measures ANOVA was conducted to
evaluate the effect of the Strong Kids curriculum over the course of 13 weeks on
internalizing symptoms associated with emotional-behavioral problems. Two dependent
measures, the ISSC and the CDI, were used to measure any effect.
ISSC. The first dependent, repeated measures variable measuring internalizing
symptoms was the mean total raw score on the ISSC. The within-subjects factor was
0
5
10
15
20
Pretest Posttest Time
Kn
ow
led
ge T
est
Raw
Sco
re
Control No Consultation Consultation
49 Time, the between-subjects factor was Group. The Time by Group interaction effect
and Group and Time main effects were tested using the multivariate criterion of Wilks’
lambda (Λ). The ANOVA revealed no significant interaction between Group and Time, Λ
= .97, F(2, 124) = 1.89, p = .16, indicating there was no difference among groups
regarding internalizing symptoms over time. Observed power using Wilks’ Lambda was
.39, p = .05, suggesting a moderate possibility of making a Type II error and failing to
reject a false null hypothesis. Because no significant interaction was determined between
Time and Group, main effects for these variables were reviewed. There was no
significant main effect for Group, F(2, 124) = .26, p = .77, and no significant main effect
for Time, F(1, 124) = 2.53, p = .12. Observed power for Time was .35, p = .05 and for
Group .09, p = .05. Low observed power, particularly for the Group variable increases the
likelihood of making a Type II error and subsequently not detecting a significant effect
when one may be present. Table 6 shows the results of the ANOVA. No post-hoc
analyses were conducted due to no significant interaction and no main effects. Figure 3
illustrates mean raw total ISSC scores for all groups across time.
50 Table 6
Mixed 2-Way ANOVA for the Effect of Group and Time on Internalizing Symptoms as Measured by ISSC Total Raw Score
Source df F η p
Between subjects
Group 2 .26 .00 .77
Error Between 124
Within subjects
Time 1 2.53 .02 .12
Group 2 .26 .00 .77
Time * Group 2 1.89 .03 .16
Error Within 124
* p < .05.
51
Note. Horizontal line at raw score = 70 illustrates raw score at or above 70 to be in the clinically significant range. Figure 3. Internalizing Symptoms as Measured by ISSC Total Mean Raw Scores
In addition to the analysis conducted on the ISSC total symptoms mean raw score
for all groups across time, analyses were conducted for the two separate factor subscale
scores for this measure, the first factor measuring negative affect or general distress and
the second factor measuring positive affect.
ISSC: Factor 1: Negative affect/general distress. A two-way, repeated measures
ANOVA was conducted to evaluate the effect of the Strong Kids curriculum on
symptoms associated with negative affect. Factor 1 was measured by summing the
individual items on the ISSC associated with this subscale, yielding a total raw score. The
within-subjects factor was Time, the between-subjects factor was Group. The Time by
Group interaction effect and Group and Time main effects were tested using the
0 10 20 30 40 50 60 70 80 90
100 110 120 130 140
Pretest Posttest Time
Mea
n R
aw S
core
on
ISSC
Control No Consultation Consultation
52 multivariate criterion of Wilks’ lambda (Λ). The ANOVA revealed no significant
interaction between Group and Time, Λ = .98, F(2, 122) = 1.39, p = .25, indicating no
difference among groups regarding feelings or behaviors associated with negative affect
or general distress over time. Observed power using Wilks’ Lambda was .29, p = .05,
suggesting a moderate to high possibility of making a Type II error and failing to reject a
false null hypothesis or detecting a significant effect. Because no significant interaction
was determined between Time and Group, main effects for these variables were
reviewed. No significant main effect for Time was found, F(1, 122) = 2.53, p =.12, and
no significant main effect for Group was found, F(2, 122) = .28, p =.76. Observed power
for Time was .31, p = .05 and for Group .09, p = .05. Low observed power, particularly
for the Group variable increases the likelihood of making a Type II error and
subsequently not detecting a significant effect when one may be present. Table 7 shows
the results of the ANOVA. No post-hoc analyses were conducted due to no significant
interaction and no main effects. Figure 4 illustrates mean ISSC Factor 1 scores for all
groups across time.
53 Table 7
Mixed 2-Way ANOVA for the Effect of Group and Time on Feelings and Behaviors Associated with Negative Affect/General Distress as Measured by ISSC: Factor 1 Scale Source df F η p
Between subjects
Group 2 .28 .01 .76
Error Between 122
Within subjects
Time 1 2.53 .02 .12
Group 2 .28 .01 .76
Time * Group 2 1.39 .02 .25
Error Within 122
* p < .05.
54
Note: Horizontal line at raw score = 70 illustrates raw score at or above 70 to be in the clinically significant range. Figure 4. Negative Affect/General Distress Symptoms as Measured by ISSC: Factor 1 Scale
ISSC: Factor 2: Positive affect. A two-way repeated measures ANOVA was
conducted to evaluate the effect of the Strong Kids curriculum on feelings and behaviors
associated with positive affect. Factor 2 was measured by summing the individual items
on the ISSC associated with this subscale, yielding a total raw score. The within-subjects
factor was Time, the between-subjects factor was Group. The Time by Group interaction
effect and Group and Time main effects were tested using the multivariate criterion of
Wilks’ lambda (Λ). The ANOVA revealed no significant interaction between Group and
Time, Λ = .99, F(2, 122) = 2.04, p = .13, indicating no difference among groups
0 10 20 30 40 50 60 70 80 90
100
Pretest Posttest Time
Tota
l Mea
n R
aw S
core
on
ISSC
N
egat
ive
Aff
ect S
cale
Control No Consultation Consultation
55 regarding feelings and behaviors associated with positive affect over time. Observed
power using Wilks’ Lambda was .41, p = .05, suggesting a moderate probability of
making a Type II error and failing to reject a false null hypothesis. Because no significant
interaction was determined between Time and Group, main effects for these variables
were reviewed. No significant main effect for Time was found, F(1, 122) = 2.58, p = .11,
and no significant main effect for Group was found, F(2, 122) = .01, p = .99. Observed
power for Time was .36, p = .05 and for Group, .05, p = .05. Low power observed for the
Group variable increases the likelihood of making a Type II error and subsequently not
detecting a significant effect for Group when one may be present. Table 8 shows the
results of the ANOVA. No post-hoc analyses were conducted due to no significant
interaction and no main effects. Figure 5 illustrates mean ISSC Factor 2 scores for all
groups across time.
56 Table 8
Mixed 2-Way ANOVA for the Effect of Group and Time on Positive Affect as Measured by ISSC: Factor 2: Positive Affect Scale
* p < .05.
Source df F η p
Between subjects
Group 2 .01 .00 .99
Error Between 122
Within subjects
Time 1 2.58 .02 .11
Group 2 .01 .00 .99
Time * Group 2 .97 .03 .13
Error Within 122
57
Note: Horizontal line at raw score = 25 illustrates raw score at or above 25 to be in the clinically significant range. Figure 5.Positive Affect as Measured by ISSC: Factor 2 Scale
CDI. The second dependent, repeated measures variable to measure internalizing
symptoms was the CDI. A two-way repeated measures ANOVA was conducted to
evaluate the effect of the Strong Kids curriculum on internalizing symptoms. The within-
subjects factor was Time and the between-subjects factor was Group. The Time x Group
interaction effect and Group and Time main effects were tested using the multivariate
criterion of Wilks’ lambda (Λ). The ANOVA revealed no significant interaction between
Group and Time, Λ = .97, F(2, 123) = 1.65, p = .20, indicating no difference among
groups regarding internalizing symptoms over time. Observed power for this non-
significant interaction, using Wilks’ Lambda was .34, p = .05, suggesting a moderate
possibility of making a Type II error and failing to reject a false null hypothesis. Because
0
10
20
30
40
50
Pretest Posttest Time
Mea
n R
aw S
core
on
the
ISSC
Po
sitiv
e A
ffect
Sca
le
Control No Consultation Consultation
58 no significant interaction was determined between Time and Group, main effects for
these variables were reviewed. No significant main effect was found for Time, F(1, 123)
= .77, p = .38, and no significant main effect was found for Group, F(2, 123) = .10, p =
.91. Observed power for Time was .14, p = .05 and for Group, .07, p = .05. Low
observed power for both variables increases the likelihood of making a Type II error and
subsequently not detecting significant effects when they may be present. Table 9 shows
the results of the ANOVA. No post-hoc analyses were conducted due to no significant
interaction and no main effects. Figure 6 illustrates mean CDI scores for all groups across
time.
Table 9
Mixed 2-Way ANOVA for Group and Time on Internalizing Symptoms as Measured by the CDI
Source df F η p
Between subjects
Group 2 .10 .00 .91
Error Between 123
Within subjects
Time 1 .77 .01 .38
Group 2 .10 .00 .91
Time * Group 2 1.65 .03 .20
Error Within 123
* p < .05.
59
Note. Horizontal line at raw score of 23 indicates clinical cutoff score for girls; raw score of 26 is clinical cutoff score for boys. Figure 6. Internalizing Symptoms as Measured by the CDI
Effect of Consultation on Knowledge, Internalizing Symptoms, and Office Disciplinary Referrals
Regarding the third research question: Are student outcomes improved when a
teacher is provided regular consultation via performance feedback from a consultant
having expertise in Strong Kids, no advantage was indicated. Although an ANOVA
conducted for a measure of knowledge of skills indicated a significant interaction effect,
a post-hoc analysis using the Bonferroni approach to control for Type I error indicated
the mean difference in total knowledge score between the group whose teacher received
consultation and the group whose teacher did not was not significant (-1.02, p > .05). The
0
10
20
30
40
50
Pretest Posttest Time
Mea
n R
aw S
core
on
the
CD
I
Control No Consultation Consultation
60 significant difference was found between the control group and the consultation group
and the control group and the no consultation group. This finding suggests that the use of
consultation did not have a statistically significant impact on student outcomes, and that
students did not necessarily benefit from receiving instruction from a teacher who had
consultative support. Because ANOVA’s conducted on measures of internalizing
symptoms and office disciplinary referrals revealed no significant interactions or main
effects, no further post-hoc calculations were performed. A review of the mean total
scores for the dependent measures for knowledge, internalizing symptoms, and office
disciplinary referrals suggests some change in a desired direction, although not
statistically significant. For example, the group whose teacher received consultation
reported slightly less internalizing symptoms on the CDI as compared to the group whose
teachers did not receive consultation. Table 10 shows the descriptive statistics between
these two groups.
61 Table 10
Descriptive Statistics for Dependent Measures at Posttest, Comparing the No Consultation Group and the Consultation Group Effect of Strong Kids on Disciplinary Referrals
Measure No Consultation Consultation
Mean SD Mean SD
Knowledge
Posttest 15.60 2.99 16.15 2.64
ISSC
Posttest 39.73 24.75 45.24 22.34
CDI
Posttest 6.50 9.36 6.00 5.89
ORD
Posttest .25 .78 .33 1.18
62 Regarding the fourth research question: What is the effect of participation in
the Strong Kids social and emotional learning curriculum on disciplinary referrals among
middle school students in a general education setting?, a two-way repeated measures
ANOVA was conducted to evaluate the effect of the Strong Kids curriculum on office
disciplinary referrals (ODRs). The total number of ODRs was calculated and the number
of major and minor infractions which comprise this total score was also calculated. Table
11 indicates the descriptive statistics for ODRs: total, minor, and major.
Table 11 Descriptive Statistics for Office Disciplinary Referral Data
Measure Control No Consultation Consultation
Mean SD Mean SD Mean SD
ODR Total
Pretest .54 1.24 .35 1.12 .67 2.76
Posttest .28 .62 .25 .78 .33 1.18
ODR Majors
Pretest .11 .60 .18 .71 .24 .98
Posttest .07 .25 .10 .50 .10 .37
ODR Minors
Pretest .39 .74 .18 .55 .43 1.80
Posttest .20 .45 .15 .43 .24 .96
63 Total ODRs. The mean total number of ODRs was tabulated and considered the
first dependent, repeated measures variable in this analysis. The within-subjects factor
was Time and the between-subjects factor was Group. The Time by Group interaction
effect and Group and Time main effects were tested using the multivariate criterion of
Wilks’ Lambda (Λ). The ANOVA revealed no significant interaction between Group and
Time, Λ = 1.00, F(2, 122) = 2.04, p = .13, indicating no difference among groups
regarding the total number of ODRs over time. Observed power using Wilks’ Lambda
was .10, p = .05, suggesting a large possibility of making a Type II error and failing to
reject a false null hypothesis. Because no significant interaction was determined between
Time and Group, main effects for these variables were reviewed. No significant main
effect was found for Time, F(1, 125) = 3.61, p = .61, and no significant main effect was
found for Group, F(2, 125) = .25, p = .78. Observed power for Time was .47, p = .05 and
for Group .09, p = .05. Low power observed for the Group variable increases the
likelihood of making a Type II error and subsequently not detecting a significant effect
for Group when one may be present. Table 12 shows the results of the ANOVA. No post-
hoc analyses were conducted due to no significant interaction and no main effects. Figure
7 illustrates mean total ODR scores for all groups across time.
64 Table 12 Mixed 2-Way ANOVA for Group and Time on Total Office Disciplinary Referrals
Source df F η p
Between subjects
Group 2 .25 .00 .78
Error Between 125
Within subjects
Time 1 3.61 .03 .61
Group 2 .25 .00 .78
Time * Group 2 .30 .01 .74
Error Within 125
* p < .05.
65
Figure 7. Mean Total Office Disciplinary Referrals
Major and minor ODRs. Two-way, repeated measures ANOVAs were used to
examine change over time among the three treatment groups. Major ODRs were rule
infractions that included non-compliant, bullying, and disruptive behavior. Minor ODRs
were rule infractions that included tardy and minor disruptive behavior and inappropriate
language. Results of this analysis yielded no significant interactions for either major
ODRs , Λ = 1.00, F(2, 125) = .20, p = .82, or for minor ODRs, Λ = .99, F(2, 125) = .65, p
= .52. Because no significant interaction effects were determined for major and minor
ODRs, main effects were examined for significant effects. There was no significant main
effect for major ORDs for Time, F(1, 125) = 1.69, p = .20, and no significant main effect
0
0.5
1
1.5
2
2.5
3
Pretest Posttest
Time
Mean
Tota
l O
ffic
e
Dis
cip
lin
ary
Refe
rrals
Control
No Consultation
Consultation
66 for Group, F(2, 125) = .31, p = .73. Although there was no main effect found for
Group for minor ODRs, F(2, 125) = .44, p = .65, a main effect was determined for Time
for minor ODRs, F(1, 125) = 4.01, p = .05.
Magnitude of Treatment Effects for Self-Report Measures and Disciplinary Referrals
Effect size (ES) calculations were conducted to determine (1) the magnitude of
the effect for dependent measures from pretest to posttest and (2) the magnitude of the
effect for same measures at posttest among treatment conditions. The standardized mean
difference procedure (Lipsey & Wilson, 2001) was used to calculate effect size. To
evaluate the magnitude of effect from pretest to posttest, the mean raw score at posttest
was subtracted from the mean raw score at pretest, this result then divided by the
harmonic standard deviation at pretest and posttest. To compare the magnitude of effect
for each dependent measure at posttest among the three treatment groups, the difference
in mean raw scores between two treatment groups (e.g., consultation vs. no consultation)
was calculated and the result then divided by the harmonic standard deviation of the two
groups at posttest. Effect sizes were evaluated for their level of significance.
Interpretation was performed by using Cohen’s (1988) recommended interpretation
guidelines: <.20 is not meaningful, .20 - .49 is considered a meaningful but small effect,
.50 - .79 is considered a medium effect, and >.80 is a large effect. For example, an ES of
.5 is interpreted as a medium effect and indicating a difference in scores at posttest
between two groups to be similar to one-half of a standard deviation unit. A positive
effect size was assigned when the calculated effect size demonstrated an effect in a
desired or anticipated direction. A negative effect size was assigned when an effect was
found to be in an undesirable or unanticipated direction.
67 Magnitude of effect from pretest to posttest. A review of effect size calculations
for dependent measures from pretest to posttest indicated five meaningful effects. For the
Control group, a small, meaningful effect was found on a measure of internalizing
symptoms (CDI, ES = .20). This is interpreted as the Control group reported an increase
in internalizing symptoms from pretest to posttest equal to two-tenths of a standard
deviation unit. A second small, meaningful negative effect was determined for the
Control group regarding total office disciplinary referrals (ODRs) indicating a decrease in
total ODRs from pretest to posttest (ES = -.28). Two meaningful effect sizes were
determined for the No Consultation group. A large, meaningful effect was found for this
group regarding knowledge of skills, indicating that this group’s knowledge increased
from pretest to posttest (Knowledge Questionnaire, ES = .89). A small, meaningful
effect was found for this group for internalizing symptoms indicating a decrease in
symptoms from pretest to posttest (ISSC Total Score, ES = .20). For the Consultation
group, a medium, meaningful effect was determined in the area of knowledge of skills,
indicating that this group’s knowledge increased from pretest to posttest (Knowledge
Questionnaire, ES = .65). Table 13 shows the results of this analysis for all groups.
68 Table 13
Magnitude of Effect from Pretest to Posttest for Dependent Measures as Measured by Effect Sizes
* = meaningful effect size.
Magnitude of effect of posttest among treatment groups. A review of effect size
calculations for dependent measures at posttest among groups indicated five meaningful
effects. The largest effect was between the Control and Consultation group on knowledge
of skills. A large, meaningful effect was found (ES = 1.32) indicating that the
Consultation group reported more knowledge of skills than the Control group at posttest.
A second large, meaningful effect was found between the Control group and the No
Consultation group (ES = 1.06) indicating the No Consultation group reported more
Dependent Measure Control No Consultation Consultation
Knowledge
Questionnaire
.07
.89* -.65*
ISSC .00 .20* .02
ISSC: Factor 1
Negative
Affect/General Distress
.00 .19 .05
ISSC: Factor 2
Positive Affect
.06 .19 .12
CDI .20* .03 0
ODR Total -.28* .11 .17
69 knowledge of skills than the Control group at posttest. A third meaningful effect was
determined to be between the No Consultation and Consultation groups in the area of
knowledge of skills (ES = .20). This effect was small, indicating the Consultation group
to have more knowledge of skills than the No Consultation group. A fourth meaningful
effect was determined between the Control group and No Consultation group on a
measure of internalizing symptoms (ISSC Total). This effect was small and negative,
indicating the No Consultation group reported fewer internalizing symptoms than the
Control group at posttest (ES = .24). The fifth meaningful effect was determined between
the No Consultation and Consultation groups also on the same measure of internalizing
symptoms. This effect was small and negative, indicating the Consultation group to have
more internalizing symptoms than the No Consultation group at posttest (ES = -.23).
Table 14 indicates the effect sizes resulting from this analysis.
70 Table 14
Magnitude of Effect at Posttest Among Treatment Groups for Self-Report Measures’ Total Scores as Measured by Effect Sizes
Note. * Denotes effect size greater than .19 and interpreted as a meaningful effect. Social Validity of the Strong Kids Curriculum
Regarding the final research question: To what extent is the Strong Kids
curriculum perceived by teachers to (a) match the goals of the curriculum (i.e., increasing
knowledge of students’ knowledge of coping strategies and decrease symptoms) to those
of the teachers’, (b) have procedures which are acceptable to the teachers, and (c) yield
teacher satisfaction with respect to observed student outcomes, a qualitative analysis was
Control Measure E.S.
No Consultation Measure E.S.
Consultation Measure E.S.
Control
Knowledge ISSC CDI ODR
NA
NA
NA
NA
Knowledge ISSC CDI ODR
1.06*
.24*
.10
.04
Knowledge ISSC CDI ODR
1.32*
.01
.18
.06
No Consultation
Knowledge ISSC CDI ODR
1.06*
.24*
.10
.04
Knowledge ISSC CDI ODR
NA
NA
NA
NA
Knowledge ISSC CDI ODR
.20*
-.23*
.07
-.08
Consultation
Knowledge ISSC CDI ODR
1.32*
.01
.18
.06
Knowledge ISSC CDI ODR
.20*
-.23*
.07
.08
Knowledge ISSC CDI ODR
NA
NA
NA
NA
71 conducted. The two teachers who implemented the curriculum completed the Strong
Kids Social Validity Survey, the responses to which were reviewed.
Alignment of goals of curriculum to those of teachers’. Both teachers were in
100% agreement in their responses to all five questions in this area. Both teachers found
the following areas to be “very important”: (1) the importance of students having
knowledge regarding coping skills for their use during difficult times and (2) the
importance of students experiencing fewer social, emotional, and behavioral problems.
Both teachers found the following areas to be “somewhat feasible/confident”: (1) the
feasibility of a teacher to instruct students on coping skills, (2) their confidence in
implementing a structured curriculum, and (3) the feasibility of a teacher providing early
intervention instruction to prevent emotional problems.
Acceptability of procedures. Both teachers were in 100% agreement in their
responses to eight of the thirteen questions in this area. Both teachers found the following
to be “very acceptable”: (1) having curriculum materials provided to them, (2) the
amount of support given by the consultant, (3) phone consultation provided by the
consultant, (4) email support provided by consultant, and (5) in-person support provided
by consultant. It is noteworthy that both teachers were provided very different levels of
support, yet found agreement among these items. Both teachers found the following to be
“not acceptable”: (1) the amount of time it took to implement each lesson, (2) the amount
of time it took to implement all lessons, and (3) the number of lessons in the curriculum.
Satisfaction with results. Both teachers were in 100% agreement in their
responses to four of the six questions in this area. Both teachers were “somewhat
satisfied” with the knowledge that students demonstrated during curriculum
72 implementation. Both teachers observed an increase in students’ knowledge. Both
teachers observed no change in students’ problem-solving skills as well as no change in
students’ demonstration of positive emotions.
Feasibility, importance and confidence in implementing curriculum. Both
teachers were in 100% agreement in their responses to two of the five questions in this
area. Both teachers felt it was “somewhat important” to implement Strong Kids. Both
teachers reported it was “somewhat feasible” for them to devote 15 minutes of prep time
to preparing for a Strong Kids’ lesson.
Internal Consistency for Dependent Measures
Although not a primary research question, internal consistency reliability
estimates were computed for the following quantitative dependent self-report measures,
as a way to ascertain their overall psychometric integrity: The Strong Kids Knowledge
Test, the Internalizing Symptoms Scale for Children (including calculations for the two
factor scores), and the Children’s Depression Inventory. Cronbach’s Alpha method was
used for this analysis. These results indicated acceptable reliability for all measures for
purposes of this research project. Reliability coefficients for each measure are shown in
Table 15.
73 Table 15
Alpha Coefficients for Quantitative Self-Report Measures at Pretest and Posttest
In addition to the internal consistency reliability analyses, Pearson product-
moment correlation coefficients were computed among the dependent self-report
measures at pretest, for the purpose of establishing convergent and discriminant validity
among these measures. The Bonferroni approach was used to control for Type I error
when calculating these correlations. A p value of less than .003 (.05/10 = .005) was
required for significance. The results of the correlational analysis indicated that 7 out of
the 10 correlations were statistically significant, 6 of these were large coefficients equal
or greater than .70. The correlations between internalizing symptoms measures (ISSC,
ISSC Factor 2, and CDI) tended to be highest. A small, negative correlation was found
between knowledge of skills and negative affect (r = -.28, p = .00) and a medium,
negative correlation was found between knowledge and overall internalizing symptoms (r
= -.37, p = .00). The results of this analysis are listed in Table 16.
Measure Pretest Posttest
Knowledge .69 .73
ISSC Total .95 .95
ISSC Factor 1 .95 .94
ISSC Factor 2 .91 .91
CDI .91 .93
74 Table 16
Correlation Coefficients Depicting Convergent and Discriminant Validity of Dependent Measures
Note. * = Correlation is significant at the .005 level using Bonferonni approach to control for Type I error (.05/10 = .005).
Dependent Measure 1 2 3 4 5
1. Knowledge Questionnaire 1 -.18
-.16 -.28* -.37*
2. ISSC -.18 1 .97* .86* .78*
3. ISSC: Factor 1
Negative Affect/General Distress
-.16 .97* 1 .75* .70*
4. ISSC: Factor 2
Positive Affect
-.28* .86* .75* 1 .85*
5. CDI -.37* .78* .70* .85* 1
75
CHAPTER V
DISCUSSION
This chapter includes a summary of the main findings of this study and a
discussion pertaining to interpretation of these findings. This discussion is organized
according to the research questions proposed. Limitations of the study are reviewed and
implications for future research and clinical applications are discussed.
Summary of Main Findings
The goal of this study was to investigate the efficacy of the Strong Kids social-
emotional learning curriculum with middle school-age students in a general education
setting. Specifically, this study examined the impact of the Strong Kids curriculum on
students’ knowledge of social-emotional concepts and skills, internalizing symptoms
associated with emotional-behavioral problems, and disciplinary referrals. This study also
examined whether consultation provided to a teacher while implementing the curriculum
contributed to improved outcomes for students as compared to implementing the
curriculum without consultation support. Finally, teachers’ attitudes regarding the
curriculum were examined using a survey.
Overall, students who participated in the curriculum demonstrated significantly
improved knowledge of social-emotional concepts and skills. Small, meaningful effects
were found for internalizing symptoms across time and between groups. Although no
significant effects were determined for total office disciplinary referrals, a significant
76 main effect was determined for minor ODRs over time. Teachers’ acceptability of and
satisfaction with the curriculum was generally positive, but mixed. Overall, teachers
tended to believe providing students with skills to cope with life’s challenges was very
important and they were agreeable to spending time preparing to deliver the curriculum.
Impact on Knowledge of Social-Emotional Concepts and Skills
Consistent with prior pilot studies evaluating the efficacy of the Strong Kids
curriculum on knowledge of healthy social-emotional skills and curriculum-related
content (Castro-Olivo, 2006; Isava, 2006; Merrell, Juskelis, Tran, & Buchanan, 2006),
results from this study indicted that students who participated in the curriculum
experienced an increase in knowledge of these concepts as compared with the students
who did not participate in the curriculum. Large, meaningful effects sizes were found at
posttest between the control group and no consultation group (ES = 1.06) and between
the control group and the consultation group (ES = 1.32) indicating a difference in the
means of the treatment and control groups at posttest to be equal to or exceed one
standard deviation. This finding indicates that not only did the students who received the
curriculum demonstrate more social-emotional knowledge than the students who did not
receive the curriculum, but they did so to a large extent. The students whose teacher
received consultation reported a higher level of knowledge gains than the students whose
teacher did not receive consultation, as evidenced by a small, meaningful effect size
determined between these groups at posttest (ES = .20). This result may indicate a slight
advantage to students whose teacher receives consultation regarding the mastery of
curriculum-related social and emotional content and general social-emotional knowledge.
77 Impact on Internalizing Symptoms Associated With Emotional-Behavioral Problems
A primary focus of prevention and early intervention services is to identify youth
who are experiencing internalizing problems and interrupt the progression by availing
proactive coping strategies. An epidemiological study by Lewinsohn, Clarke, Seeley, and
Rohde (1994) suggested the mean age of onset for a first major depressive disorder
(MDD) episode to be approximately 14.9 years of age (standard deviation of 2.8 years),
with girls experiencing a higher incidence than boys. In adults who have had an episode
of MDD, the first symptoms indicative of internalizing problems most frequently
occurred around the age of 12 years and more often in girls than in boys (Wilcox &
Anthony, 2004). Although this study did not focus on the prevalence of internalizing
symptoms among its participants or monitor their progression, the incidence of pre-
existing symptoms as well as a closer look at the incidence of symptoms according to age
and gender is noteworthy. Of the 126 student participants who completed pretest
questionnaires measuring internalizing symptoms, 9% (11 students) had total scores on
the CDI that were one to more than two standard deviations from the mean, suggesting
symptoms that were in the at-risk to clinically significant range when compared with
similar-age peers. Of these 126 students, 4% (5 students) self-reported symptoms in the
at-risk range and 5% (6 students) reported symptoms in the clinically significant range.
Further investigation of symptoms self-reported on the ISSC indicated 24% of student
participants (29 students) had total scores that were one to more than two standard
deviations from the mean. Of these 126 students, 17% (20 students) self-reported
symptoms in the at-risk range and 7% (9 students) self-reported symptoms in the
clinically significant range. For both the CDI and the ISSC, slightly more boys reported
78 internalizing symptoms than girls (55% compared with 45%, respectively).
Additionally, more eleven year olds reported internalizing symptoms than 12 year olds on
both measures (CDI: 11 year olds represented 73% of elevated scores; 12 year olds
represented 27%; ISSC: 11 year olds represented 59% of elevated scores; 12 year olds
represented 41%). When comparing these data with survival curve data from the Wilcox
and Anthony study, it appears that this sample was not necessarily experiencing
internalizing symptoms consistent with what may be expected for age and gender. In
other words, it would be expected that 12 year olds may experience more (degree
uncertain) internalizing symptoms than 11 year olds and, it may be expected that girls
experience more symptoms than boys around the age of 12 years.
Although some changes were detected among groups over time for both measures
of internalizing symptoms (the CDI and ISSC), these changes were not significant.
Further, there were no significant changes among the three treatment groups or across
time for each group. This finding was somewhat surprising given that some previous
studies have found significant improvements in internalizing symptoms over time as a
result of student participation in Strong Kids (Merrell, et al., 2006). Low observed power
for each of the measures of internalizing symptoms raises the possibility that significant
effects were not detected when they may have been present.
Although significant changes in internalizing symptoms were not determined,
small, meaningful effect sizes were determined and were consistent with previous studies
(Feuerborn, 2004; Isava, 2006; Merrell, et al., 2006). The first, between the control group
and no consultation group (ISSC Total, ES = .24) indicating the students who received
the curriculum, but whose teacher did not receive consultation, reported fewer
79 internalizing symptoms at a quarter of a standard deviation from the students who did
not receive the curriculum. A second small, meaningful change was found for the no
consultation group from pretest to posttest, as evidenced by a decrease in internalizing
symptoms (ISSC Total, ES = .20). Interestingly, a small, meaningful change was found at
posttest between the consultation and no consultation group (ISSC Total, ES = -.23)
indicating that students who participated in the curriculum and whose teacher received
consultation support reported more internalizing symptoms also, nearly a quarter of a
standard deviation more than the students who participated in the curriculum but whose
teacher did not receive consultation support. This finding was surprising since it was
hypothesized that outcomes would be better for students whose teacher received more
support implementing the curriculum. A fourth small, meaningful change was found from
pretest to posttest for the control group as participants’ internalizing symptoms increased
(CDI, ES = .20).
The two factors which comprise the total score on the ISSC were included in the
analysis to examine any effect the curriculum may have on negative affect/general
distress symptoms and perhaps more interestingly, feelings associated with the absence of
positive affect. Results from the analysis conducted for these two factors yielded no
significant results with respect to group differences, suggesting the curriculum did not
affect symptoms of general distress or feelings of positive affect.
Impact of Consultation Support on Knowledge, Symptoms, and Office Disciplinary
Referrals
The literature supports the use of performance feedback as an effective means for
improving student outcomes in the classroom (Noell, et al., 2005; Witt, et al., 1997) and
80 it is strongly recommended that teachers have regular support when implementing an
SEL program (Payton, et al., 2000). Given that no studies were found in the literature
review to have examined the effect of regular on-site consultation using performance
feedback with an SEL program, this study represents a pilot effort to determine whether
consultation provided to teachers while implementing an SEL program may contribute to
positive effects on student outcomes.
The results from this study indicate that there was very little advantage to students
who received instruction from a teacher who had consultative support via performance
feedback compared with students whose teacher implemented the curriculum
independent of support from a knowledgeable and experienced consultant. With the
exception of a small, meaningful effect for knowledge of skills between the students who
teacher received consultation and the students whose teacher did not (ES = .20), no other
comparisons indicated an advantage when a teacher received consultation. A review of
mean scores between the two groups suggest a trend in the desired direction for
symptoms and office disciplinary referrals (i.e., decrease symptoms and ODRs),
suggesting the possibility consultation may have some positive impact on these areas for
students, but not to a significant or meaningful extent for this study. As previously
indicated, a small meaningful effect was determined that indicated internalizing
symptoms were less improved at posttest for the consultation group as compared with the
no consultation group. This finding is difficult to interpret given a similar finding was not
corroborated with the second measure of internalizing symptoms, the CDI.
Although not a primary research question, it is interesting to note that while
treatment integrity data indicated both teachers maintained a high level of integrity
81 throughout curriculum implementation, the teacher who did not receive consultative
support implemented the curriculum on average at a higher rate of integrity (95%) than
the teacher who received consultation (87%). A visual analysis of these data indicates
that treatment integrity improved over time for the teacher who received consultation,
potentially suggesting benefit from receiving feedback, which may have led to
improvements throughout the course of the curriculum (Refer to Figure 1).
Treatment integrity has been linked to improved teacher performance and
subsequent student outcomes. Both teachers achieved a high level of treatment integrity
despite very different levels of support delivered; support that would presumably have
had a direct impact on the extent to which the curriculum was implemented as intended.
Anecdotal evidence suggested that both teachers were highly motivated to implement this
curriculum with high levels of integrity for the sole purpose of performing well for this
research project. Both teachers voiced concern regarding their performance and whether
they were “doing it right”. Given their desire to perform well, it is plausible that their
integrity data and in particular, the data of the teacher who was not given consultation
support, was a reflection of their attention to the major components of each lesson in the
curriculum and diligent efforts to implement the lessons as prescribed. As a result,
conclusions regarding the failure of consultation support to lead to superior outcomes for
students may be premature, and entirely dependent on this small and perhaps
unrepresentative sample of two teachers. In other words, having two highly motivated
teachers who wanted to perform well may have eliminated any effect consultation may
have had on treatment integrity and superior student outcomes. Certainly these conditions
may not exist universally as teachers possess a range of personal attributes and
82 environmental demands that can impact the extent to which they desire and are
actually able to implement this curriculum with high level of integrity. However, a high
level of treatment integrity for both treatment groups provides confidence that the results
measured at posttest are most likely related to having participated in the curriculum as
opposed to other factors that could influence scores at post-test.
Impact on Office Disciplinary Referrals
This study indicated that participation in the Strong Kids curriculum, did not
appear to have a significant impact on disciplinary problems associated with
externalizing behaviors. However, an analysis of minor ODRs (behaviors such as tardies,
minor disruptive behavior, and inappropriate language), indicated a significant change
from pretest to posttest. Interestingly, a small, meaningful effect was found for the
Control group (ES = .28), indicating a decrease in total ODRs from pretest to posttest.
Although it may be argued that participating in Strong Kids contributed to the rate of
acquired total ODRs for the participants in treatment groups to remain steady as
compared to the Control group participants whose ODRs increased, there was no
significant difference determined among these groups at posttest. Further, these results
should be interpreted with some caution. Measuring ODRs for this study was used as a
general indicator of problem behavior at school and does not represent a comprehensive
measurement of externalizing behaviors that may be measured via self-, parent-, teacher-
report, and/or observational methods. It has been determined that internalizing and
externalizing problems can co-occur (Angold, Costello, & Erkaul, 1999) and therefore it
is reasonable to inquire regarding this co-occurrence particularly for the student
participants with elevated scores on internalizing measures. Although this was not a
83 primary focus of this study, further analysis of this correlation may be valuable for
future research.
Teachers’ Attitudes Toward the Strong Kids Curriculum
Teachers’ attitudes toward the Strong Kids curriculum were positive in general,
but mixed. The area in which teachers were most closely aligned and positive in their
responses was their belief that it is very important for students to have coping skills to use
during difficult times as well as the importance for students to experience fewer social,
emotional, and behavioral problems. Consistent with quantitative measures of
knowledge, teachers observed an increase in students’ knowledge, but observed no
increase in students’ display of positive emotions or problem-solving skills. Teachers
were not satisfied with the amount of time each lesson took to implement, the number of
lessons in the curriculum, and the amount of time required to implement all lessons. This
finding is somewhat surprising given that other popular, commercially available and
widely-used social-emotional curricula have content that demands as much as or more of
a time commitment than the Strong Kids curriculum (e.g., Second Step, Steps to Respect,
The Penn Resiliency Program). However, both teachers felt that 15 minutes used in
preparation of implementing the curriculum was fairly feasible. Both teachers found the
amount of support given to them to be very acceptable regarding having materials
provided and the amount of support given by the consultant. This finding is interesting
given that teachers were given different support; one teacher received weekly support
either in person or via phone or email and the other teacher received no support during
the implementation of the curriculum.
84 Limitations
Results from this study should be interpreted in conjunction with several
limitations. Although one of primary goals of this study was to evaluate the efficacy of
the Strong Kids curriculum with students in a general education setting, doing so poses
some difficulty when measuring change in emotional and behavioral status. Although
several of the students reported experiencing internalizing symptoms measured in an at-
risk or clinically significant and elevated range, the majority of the student participants
reported symptoms within average limits. This finding should be consistent with
symptoms reported in the general population (e.g., depression symptoms in adolescents
estimated to occur in approximately 0.4% to 8.3% of adolescents every year (Greenberg,
Domitrovich, & Bumbarger, 2001)). Subsequently, although change in symptoms is
possible and desirable, the number and intensity of symptoms present at pretest may be
minimal and within normal limits for the majority of respondents. Consequently, the
likelihood of students reporting change in symptoms at posttest may be negligible and if
measured, a decrease in symptoms may not mean a change that would meaningfully alter
respondents’ mood state.
Second, low observed power was measured for all measures of internalizing
symptoms, which may potentially lead to increased chances for Type II error and
therefore not detecting change in symptoms among groups over time. For example,
observed power for detecting a significant interaction between Group and Time on the
CDI was measured to be .34 (p <.05). This would indicate that if this study was repeated,
66% of the time a false null hypothesis would not be rejected.
85 Third, as with many studies that use pre- and posttest measures to evaluate
change over time, it is possible that the pretest increased participants’ sensitivity to the
posttest. Specifically, the content of the pretests may have alerted participants to pay
closer attention to the curriculum in a way they would not have in absence of the pretests.
Consequently, participants’ scores at posttest may have been influenced by this
heightened sensitivity to possible content on the posttest that they should be paying
attention to while participating in the curriculum.
Although the number of student participants in this study was reasonable to study
the impact of the curriculum on knowledge and symptoms, the number of teacher
participants assigned to a treatment condition was minimal. A larger number of teachers
assigned to each treatment condition would increase the power to detect whether
consultation support can positively influence student outcomes.
Finally, this study was conducted in a naturalistic field setting (i.e., public middle
school, general education classroom), which is optimal for evaluating the generalizability
of this study to other similar and practical settings. Strong Kids was designed to be
implemented in classrooms and this study provides evidence for its efficacy in this
setting. Regarding demographic considerations and associated limitations, this study’s
population sample was comprised of near-equal male to female and age ratios (i.e., 55%
male, 45% female; 52% 11 years-old, 48% 12 years-old). 90% of registered students in
the school were reported as White and 33.3% of the school’s population qualified for
free-and-reduced lunch. Although the demographics of this sample are similar to that of
the school district as a whole, limitations may exist when interpreting these results in the
context of national and/or international populations with differing demographics.
86 Implications for Future Research
Although it is highly desirable that students who have participated in the Strong
Kids program learn ways in which to cope with inevitable life stressors and experience a
decrease in behaviors and feelings associated with internalizing and externalizing
problems, measuring change in symptoms appears to be a challenge with a sample of
typical students who exhibit very low base-rates of symptoms to begin with. Particularly
with the general education population, and because one of the goals of Strong Kids is to
build social and emotional resiliency, perhaps social and emotional strengths rather than
or in addition to problem symptoms should be targeted for measurement to provide a
clearer description of skills experiencing positive change. The Strong Kids curriculum
was designed to be used for preventative, inoculation efforts for youth to build resiliency
skills for use during stressful times. Measuring the accumulation of these skills as well as
the ability to apply them under duress will continue to propel this line of research
forward. This study used a simple, curriculum-linked assessment tool to measure
knowledge of social-emotional skills (the Strong Kids Knowledge Test) that focused on
the content of the curriculum and change in skill level over time as students were exposed
to curriculum content. This measurement is somewhat limited in its ability to evaluate a
curriculum that presents a number of concepts, often complex in theoretical orientation
and application. It may be helpful to expand upon this measure in an effort to more
precisely capture the breadth and depth of students’ knowledge and again, their
application of skills, in the short- and long-term. Similarly, long-term maintenance of
these skills and application is strongly desired following reception of the Strong Kids
87 curriculum. Longitudinal measurement of skills, social-emotional assets, and
internalizing and externalizing symptoms is highly desirable.
Replication and expansion upon this current study is necessary to further evaluate
the relevance and efficacy of providing consultation support to teachers. Results from this
study suggest consultation may not necessarily be needed to impact student outcomes if
an easy-to-use and brief social-emotional learning program is used. However, support has
been strongly suggested when SEL programs are used in school setting. The question for
future research will be how much and what kind of support is sufficient to efficiently
support teachers with implementation efforts and positively impact student outcomes
over the short- and long-term.
This study used a very specific type of consultation support, performance
feedback utilizing components of motivational interviewing, in an effort to combine a
strategy that has been repeatedly shown to assist teachers and positively impact student
outcomes (performance feedback) with a strategy shown to decrease resistance to change
(motivational interviewing) and subsequently, believed to decrease teacher resistance to
feedback. The use of consultation in classroom settings for research purposes is
historically fraught with implementation inconsistencies due to vaguely described
delivery approaches that are typically led by the consultant without consideration for
consultee-resistance issues. Certainly these problems interfere with our ability to not only
increase the extent to which studies may be replicated in an effort to boost generalization,
but also hinder our understanding of the mechanism used in consultation practice which
may contribute to successful outcomes. This study attempted to respond to Sheridan et
al.’s (1996) observation and request for studies in which consultation is used, procedures
88 utilized during this process should be described in detail to improve replication efforts
and an understanding of the properties of consultation that contribute to desirable and/or
undesirable outcomes. Additionally, it is hoped that future research on the effectiveness
of consultation with teachers in classroom settings will continue to consider resistance-to-
change issues as a barrier to desirable outcomes and incorporate motivational
interviewing strategies as part of the consultation process.
Social and emotional learning is associated with students’ attitudes toward school
and overall school performance (Zins, Bloodworth, et al., 2004). Although this study did
not measure students’ attitudes toward school, the curriculum, or academic success
following participation in Strong Kids, future research should consider measuring these
outcomes as part of efficacy studies. Studying these areas should provide more
information regarding the link between SEL and academic performance and engagement,
as well as students’ attitudes toward the curriculum. It is hoped that the Strong Kids
curriculum can improve students’ social-emotional resiliency, which can increase
students’ availability for learning and subsequent achievement. A positive attitude toward
the curriculum should increase the likelihood that students will actively engage in
instruction and increase the extent to which SEL concepts may be learned.
Implications for Practice
Students are learning ways in which to cope with life stressors by participating in
the Strong Kids curriculum and teachers believe it is very important for their students to
have these skills. Given teachers’ concerns regarding the amount of time required to
implement this curriculum, the probability that teachers will be able to continue to
implement the curriculum and, do so with a positive attitude, will largely depend on the
89 perceived utility and relevance of this curriculum. Additionally and perhaps more
importantly, the likelihood of teachers implementing the curriculum will also rest on
administrative and community support to implement Strong Kids and do so throughout
students’ academic careers. Administrative support and leadership in particular, are
crucial in order to provide coordinated, well-monitored, and supported implementation
efforts (Greenberg, et al., 2003). Given increasing demands on school districts and
ultimately, teachers, to produce minimal satisfactory levels of academic achievement, this
support is imperative to allow teachers the time and resources to implement an SEL
program with success and across grade-levels to maximize student outcomes. Increased
recognition of the link between social-emotional competency and academic achievement
will hopefully persuade educational leaders and community citizens to support SEL
programming as a primary means to successful academic and practical life-skills
outcomes.
At this time it is unclear whether consultation support is needed to produce
superior outcomes for students, at least with respect to an easy-to-implement SEL
curriculum such as Strong Kids. Preliminary evidence provided by this study indicates
that consultation support may not be needed for teachers to implement the curriculum at a
high level of integrity. With the exception of students in the consultation group having
slightly more knowledge skills than the no consultation group, there does not appear to be
any advantage for students’ teacher to receive consultation support. Further, both teachers
reported being satisfied with the amount of support they received during implementation
despite having distinctly different levels of support. This finding may suggest: a) teachers
vary in their desire for consultative support and/or b) the consultative support provided
90 was inconsequential to implementing the Strong Kids curriculum. The Strong Kids
curriculum was specifically designed with scripts and structured activities to reduce
teacher workload, but allowing for paraphrasing of these scripts and flexibility in chosen
activities to be relevant to students needs. This design ultimately lends itself to greater
ease of implementation. It may be that the curriculum is sufficiently straightforward for
teachers to preclude the necessity of consultation support and potentially be better
accompanied by an introductory training, as was conducted at the beginning of this study.
Certainly it seems prudent that upon introducing a new curriculum, teachers be
given support that they feel necessary to successfully implement the program and this is
clearly conceded in current literature pertaining to this issue (Greenberg, et al., 2003).
While the extent and type of support needed to best support teachers implementing
Strong Kids are unclear at this time, a recent pilot study surveying 264 elementary and
middle school teachers (Buchanan, Gueldner, and Tran, 2006) suggested teachers are
open to support through consultation (37.5% of respondents held this opinion). The
majority of teachers surveyed were willing to be observed as part of this process (61.7%)
as well as receive feedback on their performance regarding implementing an SEL
curriculum (66.3%). Recruiting a trained professional to provide consultation to teachers
can be difficult given the often overwhelming responsibilities of school personnel, as
well as teachers’ acceptability of the consultation services from a variety of qualified
professionals. Interestingly, the study by Buchanan, Gueldner, and Tran suggested
teachers were most open to consultation services from a fellow teacher (63.3% of
respondents). Teachers were then willing to received consultation from a school
counselor (53.8%), a university researcher (51%) and lastly a school psychologist
91 (48.9%). While interpretation of these findings is outside the scope of this dissertation
project, implications for clinical practice must consider the professional who will provide
consultation, should schools decide implementation of a new SEL curriculum would
benefit from consultation support.
The use of a consultant should be feasible in terms of time. Consultation ranged
from approximately 4 hours (in-service training and prep) to an average of just less than
an hour per week over the course of 12 weeks (1.5 hours on the week performance
feedback was given, .25 hour when not given). Of course, provision of this support will
depend on the other job duties of the professional lending support; however, the time
needed to provide the range of support services demonstrated in this study should be
manageable for most busy school professionals.
Conclusions
This study represents continued efforts in the area of social-emotional learning
and its efficacy in a general education setting when a promising SEL program is used. In
general, positive results were found regarding increasing students’ knowledge of social-
emotional skills and teacher acceptability for the curriculum. Although small, meaningful
effects were found in decreasing internalizing problems these were not found to be
statistically significant. Additionally, at this time it appears consultation support may be
minimally advantageous for improving students’ outcomes, both in regards to acquiring
knowledge of skills and improving internalizing and behavioral problems.
Several areas for future study would be helpful to continue the evidence-basis for
the Strong Kids curriculum and SEL in general. Such areas may include, additional
inquiry into the impact of the curriculum on skills that can be generalized from the
92 curriculum content to everyday living, the use of a strengths-based measurement
system to further assess the extent to which an SEL curriculum can provide resiliency
skills, and additional study regarding the extent to which consultation support is
necessary to successfully implement an SEL program.
Given increasing demands on today’s youth to achieve academically and the
undisputed link between social and emotional skills and academic achievement, it seems
the momentum in favor of SEL in the schools will continue. The rate of acceptance is
gaining speed, as state legislatures are recognizing the value of these skills and mandating
the inclusion of SEL curricular content throughout a typical school day’s instruction, and
school personnel believe in the importance of their students’ acquiring skills that will
buffer them during inevitable life stressors. A perhaps once lofty goal of ensuring that
social and emotional skills be made available to all students is becoming realistic and
recognized as contributing to the scholastic and life-skills development for all children.
93
APPENDIX A:
STRONG KIDS KNOWLEDGE TEST
94
Strong Kids Knowledge Test Directions: This test has 20 questions about healthy and unhealthy ways to express feelings, thoughts, and behavior. Read each question carefully and pick what you think is the best answer. TRUE-FALSE. Read each sentence. If you think it is true or mostly true, circle the T, which means “true.” If you think it is false or mostly false, circle the F, which means “false.” 1. T F Self-esteem is your feelings of worth for yourself. 2. T F When identifying a problem, it is important to describe how you feel and
then listen to how the other person says they feel. 3. T F When most people feel embarrassed, they are likely to stand tall, smile,
and talk to others. 4. T F Clenched fists and trembling or shaking hands are often signs of stress. 5. T F Your friend took the last ice cream bar at the class party and you hadn’t
gotten one yet. The best way to deal with this is to first identify how you feel, figure out if you feel comfortable or uncomfortable, and then choose 3 positive ways to express your feeling.
MULTIPLE CHOICE. Circle the letter that goes along with the best answer for each question.
6. Devin’s gym teacher tells him to try out for the basketball team. Devin thinks that he
is too short and won’t make it, so he decides to not try out for the team. What thinking error is described here? a. Binocular vision b. Black and white thinking c. Making it personal d. Fortune telling
95 7. An example of an emotion that is uncomfortable for most people is
a. Excited b. Frustrated c. Curious d. Content
8. What is an emotion? a. A thought you have about a situation b. Your inner voice inside your head c. A memory you have about something that happened to you d. A feeling that tells you something about a situation
9. Self-talk is a way to calm down after you get angry. Self-talk includes telling
yourself a. I don’t deserve this b. I should get angry when something like this happens c. I can work through this d. I need to stop getting angry so often
10. Which of the following statements best describes empathy?
a. Knowing how you are feeling b. Wondering why another person is feeling sad c. Understanding another person’s feelings d. Thinking about another person
11. What is the meaning of the thinking error dark glasses?
a. Looking at the whole picture b. Seeing only the part that makes you sad c. Trying to see things in a different way d. Thinking about only the negative or bad parts of things
12. Thinking errors occur when
a. You see things differently than what really happened b. You see both the good and bad of each situation c. You think something different than your friend d. You tell yourself you shouldn’t try to do something
96 13. Reframing is a way to
a. See the whole picture b. Think about the things that make you smile c. Think about the situation more realistically d. Think about what you will do next
14. Why would you want to know how someone else is feeling? a. So you can leave them alone when they’re angry b. To better understand that person’s feelings c. To tell other people about that person d. To act the same when you are together
15. What does the ABCDE plan for optimism help you to do?
a. Look at both sites of a situation b. View situations more positively c. Control your positive and negative thoughts d. Realize that you sometimes have no control over things
16. Conflict resolution is best described as
a. Discussing a problem until there is a winner and a loser b. Arguing with another person until they see your point and give in c. Problem-solving so you can reach an agreement d. Talking about the problem until something changes their mind
17. Which of the following is a positive way to express how scared you are for your
parents to get your report card? a. Tell them why you are scared b. Hide your report card c. Tell your parents they are expecting too much from you d. Say that your grades were bad because other kids at school distracted you
18. Why is it important to make an agreement when you are trying to solve a problem?
a. To understand what the other person is feeling b. To let the other person know what you think about the problem c. To make sure both people accept the solution to the problem d. To solve the problem more quickly
97 19. Which of the following is one of the better ways to relax when you are feeling stressed?
a. Crying b. Talking about the problem with a friend c. Complaining to your mom d. Ignore the problem
20. Which of the following is the better way to deal with feeling very angry when the
person next to you in class keeps talking and annoying you? a. Yell at them and tell the to stop b. Call out to the teacher about the student c. Take their backpack to get even d. Stop, count to ten, and try to relax
98 APPENDIX B:
CHILDREN’S DEPRESSION INVENTORY
99
100
101 APPENDIX C:
INTERNALIZING SYMPTOMS SCALE FOR CHILDREN
102
103
104 APPENDIX D:
STRONG KIDS SOCIAL VALIDITY SURVEY
105 Alignment of goals between teachers and curriculum How important do you believe it to be that students’ have knowledge regarding coping skills they can use during difficult times in their lives?
□ Very important □ Somewhat important □ Not important
How feasible do you believe it is that as a teacher, you can instruct students on these coping skills?
□ Very feasible □ Somewhat feasible □ Not feasible
Describe your level of confidence you have in implementing a structured curriculum such as Strong Kids?
□ Very confident □ Somewhat confident □ Not confident
How important is it that students experience fewer social, emotional, and behavioral problems?
□ Very important □ Somewhat important □ Not important
How feasible do you believe it is that as a teacher, you can help prevent or provide early intervention instruction in an effort to help students experience fewer emotional problems?
□ Very feasible □ Somewhat feasible □ Not feasible
Acceptability of Procedures How acceptable did you find the following procedures: Having scripted lessons
□ Not acceptable □ Somewhat acceptable □ Very acceptable
Having materials, including transparencies, in-class handouts, homework handouts, prepared and provided to you
□ Not acceptable □ Somewhat acceptable
106 □ Very acceptable
The amount of time it took to prepare for each lesson
□ Not acceptable □ Somewhat acceptable □ Very acceptable
The amount of time it took to implement each lesson □ Not acceptable □ Somewhat acceptable □ Very acceptable
The amount of time it took to teach all lessons
□ Not acceptable □ Somewhat acceptable □ Very acceptable
The number of lessons in the curriculum
□ Not acceptable □ Somewhat acceptable □ Very acceptable
Level of student interest in the lessons
□ Not acceptable □ Somewhat acceptable □ Very acceptable
The amount of support given by a consultant
□ Not acceptable □ Somewhat acceptable □ Very acceptable □ Does not apply
Phone consultation provided by a consultant
□ Not acceptable □ Somewhat acceptable □ Very acceptable □ Does not apply
Email support provided by a consultant
□ Not acceptable □ Somewhat acceptable □ Very acceptable □ Does not apply
107 Pre-service training
□ Not acceptable □ Somewhat acceptable □ Very acceptable
In-person support provided by a consultant
□ Not acceptable □ Somewhat acceptable □ Very acceptable □ Does not apply
The usefulness of the feedback provided
□ Not acceptable □ Somewhat acceptable □ Very acceptable □ Does not apply
Satisfaction with results How satisfied were you with the knowledge that students’ demonstrated during the course of implementing the program?
□ Not satisfied □ Somewhat satisfied □ Very satisfied
What kind of change did you observe in students’ knowledge? □ Decline in knowledge □ No change □ Increase in knowledge
How satisfied were you with the problem-solving skills that students’ demonstrated during the course of implementing the program?
□ Not satisfied □ Somewhat satisfied □ Very satisfied
What kind of change did you observe in students’ problem-solving skills?
□ Decline in problem-solving skills □ No change □ Increase in problem-solving skills
How satisfied were you with students’ overall demonstration of positive emotion during the course of implementing the program? □ Not satisfied
108 □ Somewhat satisfied □ Very satisfied
What kind of change did you observe in students’ demonstration of positive emotion?
□ Decline in demonstration of positive emotion □ No change □ Increase in demonstration of positive emotion
Feasibility, importance, and confidence How feasible is it to implement Strong Kids in your classroom?
□ Very feasible □ Somewhat feasible □ Not feasible
How important is it to implement Strong Kids?
□ Very important □ Somewhat important □ Not important
How feasible is it for you to devote 15 minutes of prep time to preparing for Strong Kids?
□ Very feasible □ Somewhat feasible □ Not feasible
How important is it for you to devote 15 minutes of prep time to preparing for Strong Kids?
□ Very important □ Somewhat important □ Not important
How confident did you feel in implementing the curriculum?
□ Not confident □ Somewhat confident □ Very confident
Problems, likes, and dislikes Are there any problems you encountered in implementing Strong Teens? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
109 Were there any aspects of Strong Teens and the implementation thereof that you particularly liked? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Were there any aspects of Strong Teens and the implementation thereof that you particularly disliked? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
110 APPENDIX E:
TREATMENT INTEGRITY CHECKLISTS
111 Implementation Checklist
Lesson 1: About Strong Kids: Emotional Strength Training
Observation start time: ________ I. Introduction
□ Indicated to students that new curriculum will be started □ Give examples of content that will be taught
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Pretest Assessments
□ Pretest assessments given to students □ Teacher told students that they do not need to know the answers □ Instructions are provided for the assessments
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ III. Introduction to the Topics Covered
□ Supplement 1.1 is placed on the overhead projector □ Teacher orally reviews topics on the overhead transparency
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ IV. Awareness or Disclaimer Statement
□ Teacher explains that these lessons may not be enough for all students □ Teacher indicates to students that if they are experiencing large amounts of
depression or anxiety, they should access help Circle One: Not Implemented Partially Implemented Fully Implemented
Notes: _________________________________________________________________ V. Defining Behavior Expectations
□ Teacher explains that participation in sharing personal information is voluntary □ Teacher indicates that when someone is sharing a story, everyone else should
listen respectfully □ Teacher indicates that if students do not want to share with the group, but would
like to share with the teacher, they can do so after class □ Group rules are reviewed: respect, come prepared, personal things stay in group
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ VI. Closure
□ Teacher reviews with students that they will be learning about life skills □ Teacher reminds students about class rules
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ Observation finish time: ______
112 Number of Components Implemented: Percentage Implemented:
113 Implementation Checklist
Lesson 2: Understanding Your Feelings Part 1
Observation start time: ________ I. Review
□ Reviewed previous lessons’/assignments’ main ideas (obtained 3-5 adequate ideas).
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Introduction
□ Introduced the concept of identifying comfortable and uncomfortable feelings
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ III. Name and Define Skills
□ Used supplement 2.1 as overhead transparency □ Defined: emotion, comfortable, uncomfortable □ Discussed examples of these vocabulary which aids comprehension
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ IV. Feeling Identification
□ Conveyed idea to students that identifying emotions is important so we can learn how to react positively
□ Generated list of emotions □ Identified those emotions as comfortable or uncomfortable □ Distributed Supplement 2.2 as handout □ Students discuss the work they completed on their worksheets □ Teacher asks students whether emotions were complicated to identify as
comfortable or uncomfortable □ Teacher discusses that some emotions can be both comfortable and
uncomfortable Circle One: Not Implemented Partially Implemented Fully Implemented
Notes: _________________________________________________________________ V. How Do You Feel?
□ Teacher indicates to students that they are going to discuss when you might have comfortable and uncomfortable feelings
□ Teacher gives example/s of emotion, labels it as comfortable or uncomfortable, and describes when she felt that way
□ Students do the same □ Supplement 2.3 is distributed □ Follow up discussion is conducted from Supplement 2.3
114 Circle One: Not Implemented Partially Implemented Fully Implemented
Notes: _________________________________________________________________ VI. Closure
□ Teacher reviews several main ideas from the lesson
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ VII. Homework Handout
□ Supplement 2.4 is distributed
Observation finish time: ______ Percentage of Components Implemented:
115 Implementation Checklist
Lesson 3: Understanding Your Feelings Part 2
Observation start time: ________ I. Review
□ Reviewed previous lessons’/assignments’ main ideas (obtained 3-5 adequate ideas).
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Introduction
□ Introduced the concept of expressing feelings in positive or negative ways
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ III. Identify Actions that Follow Feelings
□ Conveyed 3 of the 5 ideas listed in Activity A (bulleted items) □ Conveyed idea that we do appropriate and inappropriate things when
experiencing comfortable or uncomfortable feelings □ Discussed appropriate and inappropriate ways of expressing ideas/feelings.
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ IV. Positive and Negative Examples of Showing Feelings
□ Used Supplement 3.1 to teach appropriate ways of expressing feelings □ Used examples from supplement to generate class participation and discussion □ Used Supplement 3.2 for students to generate own examples
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ V. Practice Situations and Application
□ Used Supplement 3.3 or alternative examples to ask students to engage in exercise
□ Used Supplement 3.4 for students to view during activity to guide them □ Large group discussion of activity
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ VI. Closure
□ Teacher reviews several main ideas from the lesson
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
116 VII. Homework Handout
□ Supplement 3.5 is distributed
Observation finish time: ______ Percentage of Components Implemented:
117 Implementation Checklist
Lesson 4: Dealing with Anger
Observation start time: ________ I. Review
□ Reviewed previous lessons’/assignments’ main ideas (obtained 3-5 adequate ideas).
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Introduction
□ Introduced the concept of appropriate and inappropriate ways of expressing anger.
Circle One: Not Implemented Partially Implemented Fully Implemented
Notes: _________________________________________________________________ III. Name and Define Anger and Aggression
□ Used Supplement 4.1 as an overhead transparency □ Discussed the 5 vocabulary words on overhead □ Conveyed at least 3 of 4 main ideas under Activity B □ Asked students for examples of when they have become anger □ Indicated anger doesn’t have to lead to aggression or frustration □ Indicated anger is normal emotion, but aggression can lead to problems □ Described short and long term problems for being angry/aggressive
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ IV. Introduce Anger Model and Definitions
□ Used Supplement 4.2 as an overhead transparency □ Discussed the 6 vocabulary words on overhead
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ V. Integrate and Illustrate Anger Model
□ Used Supplement 4.3 as an overhead transparency □ Student/teacher reviews steps of anger model with corresponding scenario
components (if teacher decides to use own scenario, that’s fine) □ Teacher discusses at least 4 out of the 5 main points under Activity B
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ VI. Introduce Anger Control Skills
□ Used Supplement 4.4 as an overhead transparency
118 □ Teacher reviews each of the 4 skills with students
VII. Application of Anger Control Skills
□ Used Supplement 4.5 as in-class handout or, uses own examples to generate positive and negative examples of using skills
□ Review of negative example scenario with discussion □ Review of positive example scenario with discussion
VIII. Practice or Application
□ Students role play/discuss additional scenarios using the Anger Model □ Used Supplement 4.2 as an overhead transparency for students to reference steps
of Anger Model □ Distributed Supplement 4.4 as in-class handout for students
IX. Closure □ Teacher reviews several main ideas from the lesson
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ VII. Homework Handout
□ Supplement 4.6 is distributed
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ Observation finish time: ______ Percentage of Components Implemented:
119 Implementation Checklist
Lesson 6: Clear Thinking Part 1
Observation start time: ________ I. Review
□ Reviewed previous lessons’/assignments’ main ideas (obtained 3-5 adequate ideas).
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Introduction
□ Introduced the concept of emotions and their varying levels of intensity
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ III. Identify Intensity of Emotions, Negative Thoughts, and Common Thinking Errors
□ Used Supplement 6.1 as an overhead transparency □ Teacher model example of feeling angry and where on thermometer □ Students volunteer own examples □ Teacher indicates that thoughts can co-occur with emotions □ Students identify thoughts they had with emotions □ Used Supplement 6.2 as overhead transparency and in-class handout □ Teacher reviews each of the 6 thinking errors □ Used Supplement 6.3 as an overhead transparency □ Discussed the 6 scenarios and identified thinking errors
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
IV. Closure
□ Teacher reviews several main ideas from the lesson
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ V. Homework Handout
□ Supplement 6.4 is distributed
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ Observation finish time: ______ Percentage of Components Implemented:
120 Implementation Checklist
Lesson 8: The Power of Positive Thinking
Observation start time: ________ I. Review
□ Reviewed previous lessons’/assignments’ main ideas (obtained 3-5 adequate ideas).
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Introduction
□ Introduced the concept of positive thinking
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ III. Name and Define Skills
□ Used Supplement 8.1 as an overhead transparency □ Reviewed 4 vocabulary terms □ Conveyed at least 3 of the 4 main ideas in Activity B □ Teacher facilitates class discussion regarding negative thinking and how to look
at situations differently
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
IV. Introduce the ABCDE Learned Optimism Model
□ Used Supplement 8.2 as an overhead transparency □ Teacher introduced model, reviewing 5 parts to model
V. Integrate and Illustrate ABCDE Learned Optimism Model □ Used Supplement 8.3 as an overhead transparency □ Reviewed cartoon example with the class □ Reviewed parts A,B,C with the class in more detail (asked them prompt
questions such as those on p. 100) □ Discussed additional thoughts and feelings students may have had to scenario □ Discussed part D with the class to generate alternative ways to look at the
situation □ Discussed part E with the class to determine how case example would feel if
looked at situation differently □ Generated new situations relevant to the students, using the ABCDE model to
guide positive thinking
VI. Closure □ Teacher reviews several main ideas from the lesson □ Used Supplement 8.4 to guide discussion
121 Circle One: Not Implemented Partially Implemented Fully Implemented
Notes: _________________________________________________________________ VII. Homework Handout
□ Supplement 8.2 is distributed
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ Observation finish time: ______ Percentage of Components Implemented:
122 Implementation Checklist
Lesson 11: Behavior Change: Setting Goals and Staying Active
Observation start time: ________ I. Review
□ Reviewed previous lessons’/assignments’ main ideas (obtained 3-5 adequate ideas).
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
II. Introduction
□ Introduced the concept of goal setting and creating an action plan
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ III. Name and Define Skills
□ Used Supplement 11.1 as an overhead transparency □ Reviewed 3 vocabulary terms □ Provided/discussed at least one example of goal setting □ Provided/discussed at least one non-example of goal setting
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________
IV. Steps to Goal Attainment
□ Conveyed at least 3 of the 4 main ideas (bulleted items before Activity A) □ Used Supplement 11.2 as an overhead transparency and in-class handout □ Reviewed 6 steps of goal attainment □ Students generate their own goals using steps to goal attainment □ Used Supplement 11.1 as an overhead transparency to summarize 6 steps of goal
attainment
V. Closure □ Teacher reviews several main ideas from the lesson
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ VI. Homework Handout
□ Supplement 11.3 is distributed
Circle One: Not Implemented Partially Implemented Fully Implemented Notes: _________________________________________________________________ Observation finish time: ______ Percentage of Components Implemented:
123
APPENDIX F:
GRAPH TO TRACK INTEGRITY DATA
124
% of Components Implemented
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8
Session #
Percen
tag
e
125 APPENDIX G:
RECRUITMENT LETTER
126
Recruitment Script: Administrator
“Barbara Gueldner, a current doctoral student at the University of Oregon in the school psychology program, will be conducting a research project this coming spring, 2006. She is looking for teacher volunteers to participate, along with the students in those classrooms. Three teachers are needed to participate in this project. Participation would include teaching a social and emotional learning curriculum intended to help teach students coping skills to deal with everyday life issues. If you are interested in participating, please contact me as soon as possible. Thank you.”
127 APPENDIX H:
GENERAL INFORMATION HANDOUT
128
Strong Kids
What is it? • Developed by the UO School Psychology Program’s Oregon Resiliency Project. Please
visit us at: http://orp.uoregon.edu/ • 12 lesson social and emotional learning curriculum intended to help kids cope with
difficulties in life • Intended to help build resiliency skills to prevent depression and anxiety symptoms • Lesson’s include: understanding your feelings, understanding other people’s feelings
(empathy) dealing with anger, learning strategies to think more clearly about situations (learned optimism), conflict resolution training, coping with stress, setting goals, and positive thinking strategies
Why am I doing this?
• This is a study for the purposes of dissertation research. Your participation will not only assist the student researcher in fulfilling the dissertation portion of the doctoral degree requirement, but move the field of prevention and early intervention science forward by empirically validating the effectiveness of the Strong Kids’ curriculum.
What my role and responsibilities?
• 2 teachers will be selected to teach the lessons in their classroom during the course of a typical class period.
• A 3rd teacher will not teach the lessons, but will administer pre and post test measures in class (this will take approximately 20 minutes over the course of 2 class periods).
• The lessons are intended to be taught once or twice per week; if taught once per week, you would teach the lessons over the course of 12 weeks; if taught twice per week, you would teach the lessons over the course of 6 weeks.
How am I going to do that?
• Teachers will receive a 2 hour training to review materials and address questions or concerns.
• Teachers who deliver the curriculum will receive assistance from the student researcher. • All materials will be provided to you before you begin the first lesson.
What do we expect students to get out of this?
• Prior pilot studies have indicated that students gain in knowledge about ways to cope with difficulties and show a decrease in anxiety and depression symptoms.
• We also expect students to have fun!
What can I expect to get out of this? • We hope a relatively simple, useful, and meaningful curriculum to teach your students
social and emotional skills. • A thank you gift for your participation!
129 Contacts if I have questions:
• Barbara Gueldner, M.S.E., Student Researcher, 729-4211 or [email protected]
• Ken Merrell, Ph.D., Dissertation Committee Chair, 346-2412
130 APPENDIX I:
PARENT CONSENT LETTER, TREATMENT GROUPS
131
February 3, 2006 Dear Parent/Legal Guardian, Your child’s school, Thurston Middle School, has adopted a curriculum called Strong Kids, a program designed by the University of Oregon to build resiliency skills by teaching students how to handle typical stress and social situations in a positive manner. Resiliency skills are the skills that students use everyday to overcome minor problems in their environment. Since resiliency is the ability to bounce back, some of the skills covered in the resiliency program will be problem-solving, positive-thinking, goal-setting, and anger-management.1 This curriculum will begin in approximately two weeks. Thurston Middle School has volunteered to be a part of a research study to evaluate the effectiveness of this curriculum. This study is being conducted by Barbara Gueldner, M.S.E., a doctoral student at the University of Oregon and supervised by Dr. Ken Merrell, the director of the School Psychology Program at the University of Oregon. Your child was selected as a possible volunteer because he/she will be receiving this curriculum as part of the general education, language arts curriculum and his or her teacher has been trained to present these lessons. The lessons will be presented in approximately 50-minute sessions once a week for twelve weeks during a regularly scheduled language arts class. It is anticipated that students will learn social and emotional strategies to foster resiliency and prevent social, emotional, and behavioral problems. To check on the effectiveness of the resiliency lessons, your child will be given three short questionnaires before the lessons are presented and then three more short questionnaires at the end of the twelve weeks. Participation is voluntary. Each questionnaire will take approximately 10 minutes to complete. The questionnaires are easy to complete and will ask questions about their feelings about themselves, their relationships, and their abilities. The students are given these tests at the end of the twelve weeks to see if the lessons were effective in teaching resiliency skills. There is no grade attached to your student’s performance on these tests or for their performance throughout the twelve lessons. In addition, school behavioral records will be reviewed to evaluate the effect this curriculum may have on student behavior. Your child’s name will not be associated with any of the information gathered (the questionnaires and behavioral records).
132 The questionnaires that your child will be asked to complete are of minimal psychological risk. Responding to questions regarding feelings could possibly be unpleasant or mildly upsetting to students. Your child’s teacher is trained to monitor these situations closely and to anticipate concerns that may be unique to his or her students. The researcher will also be monitoring these procedures. To respect your child’s privacy, any written information will be given a code and will not be attached to his or her name. All of the coded information will be kept at the University of Oregon, and only general information like age, grade, gender, and ethnicity (if provided) will be attached to the code. Your decision whether or not to participate will not affect your relationship with your child’s district, school, teacher, or with the University of Oregon. If you decide that your child will not participate in the assessment sessions, a supervised and structured activity will be provided for your child. Because each of the 2 assessment sessions is expected to last only 20 - 30 minutes, the activity will most likely be in the form of a structured study session. If you decide to participate, you may still withdraw your consent and discontinue your child’s participation at any time without penalty. If you have any questions, please feel free to contact Barbara Gueldner at 541.346.2680 or Dr. Ken Merrell at 541.346.2414. If you have questions regarding your or your child’s rights as a research participant, contact the Office of Human Subjects Compliance, University of Oregon, Eugene OR 97403, (541) 346-2510. You will be given a copy of this form to keep. Receipt of this letter indicates that you have read and understood the information provided above, that you willingly agree that your child may participate, that you know that you may withdraw your consent at any time and discontinue participation without penalty, that you will receive a copy of this form, and that you are not waiving any legal claims, rights or remedies. If you decide that do not want your child to participate in this study, please call 744.6368, ask for Elaine Lessar and indicate that you do not want your child to participate in this study. Sincerely, Barbara Gueldner, M.S.E. University of Oregon 1 To view the materials that will be presented to your child or to learn more about the curriculum prior to making a decision to participate please log on to http://orp.uoregon.edu or, view a copy of the curriculum available in the school’s main office.
133 School Psychology Doctoral Candidate
134 APPENDIX J:
PARENT CONSENT LETTER, CONTROL GROUP
135 February 3, 2006 Dear Parent/Legal Guardian, Your child’s school, Thurston Middle School, has adopted a curriculum called Strong Kids, a program designed by the University of Oregon to build resiliency skills by teaching students how to handle typical stress and social situations in a positive manner. Resiliency skills are the skills that students use everyday to overcome minor problems in their environment. Since resiliency is the ability to bounce back, some of the skills covered in the resiliency program will be problem-solving, positive-thinking, goal-setting, and anger-management.2 It is anticipated that students will learn social and emotional strategies to foster resiliency and prevent social, emotional, and behavioral problems. This curriculum will begin in approximately two weeks. Thurston Middle School has volunteered to be a part of a research study to evaluate the effectiveness of this curriculum. This study is being conducted by Barbara Gueldner, M.S.E., a doctoral student at the University of Oregon and supervised by Dr. Ken Merrell, the director of the School Psychology Program at the University of Oregon. Although your child will not participate in this curriculum at this time, your child has been selected as a possible volunteer for this study to help us evaluate the effectiveness of the lessons. To check on the effectiveness of the resiliency lessons, your child will be given three short questionnaires at the beginning of this study, in approximately two weeks, and then three more short questionnaires at the end of twelve weeks. Participation is voluntary. Each questionnaire will take approximately 10 minutes to complete. The questionnaires are easy to complete and will ask questions about their feelings about themselves, their relationships, and their abilities. There is no grade attached to your student’s performance on these tests. In addition, school behavioral records will be reviewed to evaluate the effect this curriculum may have on student behavior. Your child’s name will not be associated with any of the information gathered (the questionnaires and behavioral records). The questionnaires that your child will be asked to complete are of minimal psychological risk. Responding to questions regarding feelings could possibly be unpleasant or mildly upsetting to students. Your child’s teacher is trained to monitor these situations closely and to anticipate concerns that may be unique to his or her students. The researcher will also be monitoring these procedures.
136 To respect your child’s privacy, any written information will be given a code and will not be attached to his or her name. All of the coded information will be kept at the University of Oregon, and only general information like age, grade, gender, and ethnicity (if provided) will be attached to the code. Your decision whether or not to participate will not affect your relationship with your child’s district, school, teacher, or with the University of Oregon. If you decide that your child will not participate in the assessment sessions, a supervised and structured activity will be provided for your child. Because each of the 2 assessment sessions is expected to last only 20-30 minutes, the activity will most likely be in the form of a structured study session. If you decide to participate, you may still withdraw your consent and discontinue your child’s participation at any time without penalty. If you have any questions, please feel free to contact Barbara Gueldner at 541.346.2680 or Dr. Ken Merrell at 541.346.2414. If you have questions regarding your or your child’s rights as a research participant, contact the Office of Human Subjects Compliance, University of Oregon, Eugene OR 97403, (541) 346-2510. You will be given a copy of this form to keep. Receipt of this letter indicates that you have read and understood the information provided above, that you willingly agree that your child may participate, that you know that you may withdraw your consent at any time and discontinue participation without penalty, that you will receive a copy of this form, and that you are not waiving any legal claims, rights or remedies. If you decide that do not want your child to participate in this study, please call 744.6368, ask for Elaine Lessar and indicate that you do not want your child to participate in this study. Sincerely, Barbara Gueldner, M.S.E. University of Oregon School Psychology Doctoral Candidate
2 To view the materials that will be presented to your child or to learn more about the curriculum prior to making a decision to participate please log on to http://orp.uoregon.edu or, view a copy of the curriculum available in the school’s main office.
137 APPENDIX K:
STUDENT ASSENT, CONTROL GROUP
138
Dear Student: I am a student at the University of Oregon. I am interested in helping kids like you to stay strong even when upsetting or difficult things happen in your life. I have done a lot of work to find out what helps students to stay strong when things go wrong and have figured out some of the best things that help. Your teacher, Mr. Thorsby, has read our materials agrees that these are some good things to help kids to stay strong, and he would like to help me to find out the best way to teach these things to students in your grade. In order to find out the best way to teach kids to stay strong, Mr. Thorsby is going to give you three tests to find out how much you already know about what makes you feel strong. Then, after about 12 weeks, he’ll give you three more easy tests that take about 10 – 15 minutes. Filling out these tests should help us to understand what you need to know to deal with life’s problems and stay strong. We don’t think that the questions you are asked to answer will bother you, but some of the questions ask you about your feelings and what you would do in possible life situations, such as what to do if you are angry or stressed. Mr. Thorsby has been trained to make sure that even these examples about things going wrong don’t bring up any bad feelings for you, and Mr. Thorsby will help you to remember that the situations are not real. We can help you with any bad feelings or problems that may come up after filling out these questionnaires. Your parents have already told us that it is alright if you take these tests. You will not receive any money for filling out the questionnaires, but we would still like you to complete them. You do not have to fill out the questionnaires and if you decide not to, you will not get into any trouble. If you decide that you will fill them out, just sign your name on the line below. Even if you sign, if you change your mind later on, just let the teacher or your parent know that you don’t want to complete the questionnaires, and you won’t get in any trouble for changing your mind. Remember, that completing these questionnaires will happen during the school day, not before school or after school, and the scores you get on them are not counted on your report cards. In fact, no one knows whose work it is. We will use a code name instead of your name and the code name will only tell us if you are a girl or a boy, and what grade you are in, what age you are, and maybe what race you are (if you decide to say so). If you are thinking about signing but still don’t feel sure what this is asking about, ask your parents about it, or ask if you can log onto http://orp.uoregon.edu on the internet to learn more, or you can call me, Barbara Gueldner, at my office at the University of Oregon: 541-346-2680 or Professor Ken Merrell at 541-346-2414. You will get a copy of this letter to keep and take home.
139 Sincerely, Barbara Gueldner, M.S.E. I, _________________________________, have decided to take part in this project. Signature
140 APPENDIX L :
STUDENT ASSENT, NO CONSULTATION GROUP
141
Dear Student: I am a student at the University of Oregon. I am interested in helping kids like you to stay strong even when upsetting or difficult things happen in your life. I have done a lot of work to find out what helps students to stay strong when things go wrong and have figured out some of the best things that help. Your teacher, Ms. Copeland, has read our materials agrees that these are some good things to help kids to stay strong, and she would like to help me to find out the best way to teach these things to students in your grade. For the next twelve weeks, Ms. Copeland is going to teach lessons once a week about some of the important things that we are interested in, like the best thing to do when you feel angry or sad. Before Ms. Copeland starts to teach these lessons, she is going to give you three tests to find out how much you already know about what makes you feel strong. Then, at the very end of the twelve weeks, she’ll give you three more easy tests that take about 10 – 15 minutes each and find out what you have learned. Filling out these tests should help us to understand how well the lessons help you learn skills to deal with life’s problems and stay strong. We don’t think that the questions you are asked to answer will bother you, but some of the questions ask you about your feelings and what you would do in possible life situations, such as what to do if you are angry or stressed. Ms. Copeland has been trained to make sure that even these examples about things going wrong don’t bring up any bad feelings for you, and Ms. Copeland will help you to remember that the situations are not real. We can help you with any bad feelings or problems that may come up after filling out these questionnaires. Your parents have already told us that it is alright if you have these lessons. You will not receive any money for filling out the questionnaires, but we would still like you to complete them. You do not have to fill out the questionnaires and if you decide not to, you will not get into any trouble. If you decide that you will fill them out, just sign your name on the line below. Even if you sign, if you change your mind later on, just let the teacher or your parent know that you don’t want to complete the questionnaires, and you won’t get in any trouble for changing your mind. Remember, that completing these questionnaires will happen during the school day, not before school or after school, and the scores you get on them are not counted on your report cards. In fact all of the work that you do in this class will be kept confidential so that no one knows whose work it is. We will use a code name instead of your name and the code name will only tell us if you are a girl or a boy, and what grade you are in, what age you are, and maybe what race you are (if you decide to say so).
142 If you are thinking about signing but still don’t feel sure what this is asking about, ask your parents about it, or ask if you can log onto http://orp.uoregon.edu on the internet to learn more, or you can call me, Barbara Gueldner, at my office at the University of Oregon: 541-346-2680 or Professor Ken Merrell at 541-346-2414. You will get a copy of this letter to keep and take home. Sincerely, Barbara Gueldner, M.S.E. I, _________________________________, have decided to take part in this project. Signature
143 APPENDIX M:
STUDENT ASSENT, CONSULTATION GROUP
144
Dear Student: I am a student at the University of Oregon. I am interested in helping kids like you to stay strong even when upsetting or difficult things happen in your life. I have done a lot of work to find out what helps students to stay strong when things go wrong and have figured out some of the best things that help. Your teacher, Ms. Sedlack, has read our materials agrees that these are some good things to help kids to stay strong, and she would like to help me to find out the best way to teach these things to students in your grade. For the next twelve weeks, Ms. Sedlack is going to teach lessons once a week about some of the important things that we are interested in, like the best thing to do when you feel angry or sad. Before Ms. Sedlack starts to teach these lessons, she is going to give you three tests to find out how much you already know about what makes you feel strong. Then, at the very end of the twelve weeks, she’ll give you three more easy tests that take about 10 – 15 minutes each and find out what you have learned. Filling out these tests should help us to understand how well the lessons help you learn skills to deal with life’s problems and stay strong. We don’t think that the questions you are asked to answer will bother you, but some of the questions ask you about your feelings and what you would do in possible life situations, such as what to do if you are angry or stressed. Ms. Sedlack has been trained to make sure that even these examples about things going wrong don’t bring up any bad feelings for you, and Ms. Sedlack will help you to remember that the situations are not real. We can help you with any bad feelings or problems that may come up after filling out these questionnaires. Your parents have already told us that it is alright if you have these lessons. You will not receive any money for filling out the questionnaires, but we would still like you to complete them. You do not have to fill out the questionnaires and if you decide not to, you will not get into any trouble. If you decide that you will fill them out, just sign your name on the line below. Even if you sign, if you change your mind later on, just let the teacher or your parent know that you don’t want to complete the questionnaires, and you won’t get in any trouble for changing your mind. Remember, that completing these questionnaires will happen during the school day, not before school or after school, and the scores you get on them are not counted on your report cards. In fact all of the work that you do in this class will be kept confidential so that no one knows whose work it is. We will use a code name instead of your name and the code name will only tell us if you are a girl or a boy, and what grade you are in, what age you are, and maybe what race you are (if you decide to say so).
145 If you are thinking about signing but still don’t feel sure what this is asking about, ask your parents about it, or ask if you can log onto http://orp.uoregon.edu on the internet to learn more, or you can call me, Barbara Gueldner, at my office at the University of Oregon: 541-346-2680 or Professor Ken Merrell at 541-346-2414. You will get a copy of this letter to keep and take home. Sincerely, Barbara Gueldner, M.S.E. I, _________________________________, have decided to take part in this project. Signature
146 APPENDIX N:
TEACHER CONSENT
147
February 6, 2006 Dear Teacher: Your school has agreed to participate in a research study on a resiliency curriculum conducted by Barbara Gueldner, a doctoral student in the School Psychology Program at the University of Oregon, supervised by Ken Merrell, Ph.D., Director of the School Psychology Program. Resiliency is the capacity to bounce back when presented with life-stressors, and a child’s possession of resiliency characteristics is related to positive life-outcomes. The study will investigate how a twelve-week resiliency curriculum, Strong Kids, can be incorporated into a school’s overall curriculum, and how students respond to a school-based curriculum that teaches skills such as problem-solving, positive-thinking, goal-setting, and anger-management. We would like to assess whether or not students feel that they learn new skills or knowledge after twelve-weeks of instruction in this curriculum. The study will also investigate how support and training will help teachers implement the curriculum as well as teachers’ overall perception of the curriculum. You were selected as a possible participant in this study because the principal of your school suggested that you would be willing to learn more about this type of curriculum and be a part of this study. If you decide to participate3, I will be conducting a 2 hour in-service teacher training. The training will involve instruction regarding the curriculum and the age and grade specific requirements for its presentation. Once you are trained, class-time will be scheduled to deliver the curriculum and consent forms will be provided to parents to gain permission for their students to participate in an in-school research study. The impact of the curriculum will be 45-50 minutes a week for 12 weeks. At the discretion of the principal or other decision maker, the curriculum will be presented in lieu of a language arts or related class. For the purposes of the research, you will be asked to assess students at the beginning of the curriculum and at the end of the twelve week course. The assessment will consist of three easy tests that the students fill out themselves. The tests ask simple questions about their feelings about themselves, their relationships, and their abilities and take 10 – 15 minutes each. The scores from these tests will be used to determine the curriculum’s impact on students’ knowledge of resilience, and on their resilience skills. Two of the three assessments that will be used are the Internalizing Symptoms Scale Adolescents/Children (ISSA/ISSC) and the
148 Children’s Depression Inventory (CDI). If you are not already familiar with the ISSA/ISSC and CDI, you will be trained to administer these simple measures. These will be used as a validation tool to determine how closely aligned the Strong Kids/Teens curricula are to tools currently being used for the same purposes. As part of this study you will also be observed by a university researcher during instruction time of the lessons and participate in brief feedback sessions. Finally, you will be asked to complete a brief questionnaire at the end of program. The questionnaires that you will be asked to administer to students, the observation and feedback sessions that you participate in, and the questionnaire you will be asked to complete are of minimal psychological risk. Responding to questions regarding feelings could possibly be unpleasant or mildly upsetting to students. The university investigator will monitor this procedure and will respond as appropriately. The presence of an observer in the classroom, participating in the feedback sessions, and responding to the teacher questionnaire could possibly be unpleasant. The university researcher is trained to monitor these situations closely and respond as appropriate. To maintain the anonymity of participants, any written information that is obtained in connection with this study will be securely coded and only demographic information, such as gender, years of teaching, and subject area taught will be attached to the codes. Participation of districts, schools, teachers, and students is voluntary. Your decision whether or not to participate will not affect your relationship with the University of Oregon, the Department of School Psychology, your school, or the school district. If you decide to participate, you are free to withdraw your consent and discontinue participation at any time without penalty. If you have any questions, please feel free to contact Barbara Gueldner at (51) 346-2680 or Dr. Ken Merrell at (541)346-2414. If you have questions regarding your rights as a research participant, contact the Office of Human Subjects Compliance, University of Oregon, Eugene OR 97403, (541) 346-2510. You will be given a copy of this form to keep. Your signature indicates that you have read and understand the information provided above, that you willingly agree to participate, that you may withdraw your consent at any time and discontinue participation without penalty, that you will receive a copy of this form, and that you are not waiving any legal claims, rights or remedies. ___________________________________________
3 School and district decision-makers are welcome to view the curriculum materials prior to making a decision to participate. For a general overview of the scope and nature of the curriculum log on to http://orp.uoregon.edu
149 Print Name and Title ___________________________________________ School/Grade(s) ___________________________________________ Signature and date
150 APPENDIX O:
TRAINING AGENDA
151
Agenda for Strong Kids Training A. Training with Group 1, Group 2, Control Group Teachers (15-20 minutes) 1. Review conceptual framework for curriculum Resiliency How Strong Kids promotes resiliency 2. Pre/post tests Review procedures B. Training with Group 1 and 2 Teachers (60-90 minutes) 1. General overview of all lessons Scripts, transparencies, in-class and homework handouts 2. Demonstrate and practice lessons 1 and 2 3. Review procedures Performance feedback and no performance feedback groups 4. Scheduling Lesson delivery Performance feedback sessions 5. Questions and answer session
152 APPENDIX P:
FREQUENTLY ASKED QUESTIONS HANDOUT
153 Frequently Asked Questions Has this curriculum been used before with middle school students? Yes! And with successful outcomes. What do I need to do to get started? Make sure you have reviewed the lesson before you teach it, verify you have all the handouts, and if you want to make any modifications to the lesson. Do I have to follow the scripts exactly? No. You can modify the language as you feel your students would understand the content described in the scripts. Can I skip sections? No, not for the purposes of this study. Can I divide a lesson into two parts? No, not for the purposes of this study. How should I group my students for in-class activities? Group your students in a way that maintains a balance between the students learning the content and good classroom behavior management practices. How do I work the homework assignments into the class time? Some teachers have found it helpful for students to complete or at least start the homework assignment in class. Some teachers also use homework in the review section of a new lesson to go over previously learned concepts. Can I use situations that are currently happening in my classroom to illustrate concepts in the lessons? Yes. Examples are provided in the curriculum, but you can make up your own. What do I do if a student is experiencing emotional difficulties during the course of a lesson? If you believe a student is having a difficult time with the curriculum content, becomes upset or seems distressed, please do the following:
• Notify Katie Corwin or Craig Thorne • Notify Barbara Gueldner, 729-4211
154 APPENDIX Q:
PERFORMANCE FEEDBACK CHECKLIST
155 Performance Feedback Procedures Checklist
Lesson Date PFB Given?
Review Integrity Data?
Graph Data
Review Graph w/ Teacher
3 Praise Statemt Given
Menu of Option Discuss
Resource Needed/ Barriers Discuss
Goal Agreed Upon
Goal is:
Y/N
Y/N Y/N Y/N Y/N Y/N Y/N Y/N
1
2
3
4
5
6
7
8
9
10
11
12
156 APPENDIX R:
PHONE CHECK-IN SHEET
157
Performance Feedback Group Phone Check In
Date Email or Phone Contact?
Question/ Concerns Addressed?
Discussed Ways in Which Questions Could be Addressed?
Verify Next Check In Day?
Y/N
Y/N
Y/N
Y/N
February
February
March
March
April
April
May
May
158
BIBLIOGRAPHY
Alvero, A. M., Bucklin, B. R., & Austin, J. (2001). An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings. Journal of Organizational Behavior Management, 21, 3-29.
Ager, A. & O’May, F. (2001). Issues in the definition and implementation of best
practice for staff delivery of interventions for challenging behavior. Journal of Intellectual and Developmental Disability, 26, 243-256.
Angold, A., Costello, E. J., & Erkanli, A. (1999). Comorbidity. Journal of Child
Psychology and Psychiatry, 40, 57-87. Asarnow, J. R., Jaycox, L. H., & Tompson, M. C. (2001). Depression in youth:
Psychosocial interventions. Journal of Clinical Child Psychology, 30, 33-47. Balcazar, F., Hopkins, B. L., & Suarez, Y. (1985). A critical, objective review of
performance feedback. Journal of Organizational Behavior Management, 7, 65-89.
Botvin, G. J., Baker, E., Dusenbury, L., Tortu, S., & Botvin, E. (1990). Preventing
adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology, 58, 437-446.
Brown, J. L., Roderick, T., Lantieri, L., & Aber, J. L. (2004). The resolving conflict
creatively program: A school-based social and emotional learning program. In J. E. Zins, R. P., Weissberg, M. C. Wang, & H. J. Walberg (Eds.), Building academic success on social and emotional learning: What does the research say? New York: Teachers College.
Brown, J. M. & Miller, W. R. (1993). Impact of motivational interviewing on
participation and outcome of residential alcoholism treatment. Psychology of Addictive Behaviors, 7, 211-218.
Buchanan, R., Gueldner, B. A., & Tran, O. K. (2006). Teachers’ knowledge and practices
of social and emotional learning in classrooms. Unpublished manuscript.
159 CASEL. (2003). Safe and sound. An educational leaders’ guide to evidence-based
social and emotional learning (SEL) programs. Castro-Olivo, S. M. (2006). The effects of a culturally-adapted social-emotional learning
curriculum on social-emotional and academic outcomes of Latino immigrant high school students. Unpublished doctoral dissertation, University of Oregon, Eugene.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd). Hillsdale,
NJ: Lawrence Earlbaum Associates.
Coie, J. D., Miller-Jackson, S., & Bagwell, C. (2000). Prevention science. In A. J.
Sameroff, M. Lewis, & S. M. Miller (Eds.). Handbook of Developmental Psychopathology (pp.93-108). New York: Kluwer Academic/Plenum Publishers.
Connell, D. B., Turner, R. R., & Mason, E. F. (1985). Summary of the findings of the
School Health Education Evaluation: Health promotion effectiveness, implementation, and costs. Journal of School Health, 55, 316-323.
Cossairt, A., Hall, V., & Hopkins, B. L. (1973). The effects of experimenter’s
instructions, feedback, and praise on teacher praise and student attending behavior. Journal of Applied Behavior Analysis, 6, 89-100.
Cowen, E. L. (1994). The enhancement of psychological wellness: Challenges and opportunities. American Journal of Community Psychology, 22, 149-179. Doll, B. & Lyon, M. A. (1998). Risk and Resilience: Implications for the delivery of
educational and mental health services in schools. School Psychology Review, 27, 348-364.
Elias, M.J. (2004). Strategies to infuse social and emotional learning into academics. In
In J. E. Zins, R. P. Weissberg, M. C. Wang, & H. J. Walberg (Eds.), Building academic success on social and emotional learning: What does the research say? New York: Teachers College.
Errecart, M. T., Walberg, H. J., Ross, J. G., Gold, R. S., Fielder, J. L. & Kolbe, L. J.
(1991). Effectiveness of teenage health teaching modules. Journal of School health, 61, 26-30.
Feuerborn, L. L. (2004). Promoting emotional resiliency through instruction: The effects
of a classroom-based prevention program. Unpublished doctoral dissertation, University of Oregon, Eugene.
Fixsen, D.L., Naoom, S.F., Blasé, K.A., Friedman, R.M., & Wallace, F. (2005).
Implementation research: A synthesis of the literature. National Implementation Research Network. http://nirn.fmhi.usf.edu downloaded July 20, 2005.
160 Fuchs, D. & Fuchs, L.S. (1996). Consultation as a technology and the politics of
school reform. Remedial and Special Education, 17, 386-392. Gersten, R., Chard, D., & Baker, S. (2000). Factors enhancing sustained use of research-
based instructional practices. Journal of Learning Disabilities, 33, 445-457. Greenberg, M. T., Domitrovich, C., & Bumbarger, B. (2001). The prevention of mental
disorders in school-aged children: Current state of the field. Prevention and Treatment, 4.
Greenberg, M. T., Domitrovich, C. E., Graczyk, P. A., & Zins, J. E. The study of
implementation in school-based preventative internventions: Theory, research, and practice. Retrieved electronically April 28, 2006 from www.samhsa.gov
Greenberg, M. T., Kusche, C. A., Cook, E. T., & Quamma, J. P. (1995). Promoting
emotional competence in school-aged children: The effects of the PATHS curriculum. Development and Psychopathology, 7, 117-136.
Greenberg, M.T., Weissberg, R. P., O’Brien, M. T., Zins, J. E., Fredericks, L., Resnik,
H., & Elias, M. J. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58, 466-474.
Gresham, F.M. (1989). Assessment of treatment integrity in school consultation and
prereferral intervention. School Psychology Review, 18, 37-50. Harcourt Assessment. (n.d.). Beck Depression Inventory-II (BDI-II). Retrieved March 8,
2005, from http://mail.med.upenn.edu/~abeck/scales/bdisum.html Henggeler, S.W., Melton, G.B., Brondino, M.J., Scherer, D.G., & Hanley, J.H. (1997).
Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821-833.
Hoagwood, K. & Erwin, H. D. (1997). Effectiveness of school-based mental health
services for children: A 10-year research review. Journal of Child and Family Studies, 6, 435-451.
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2004). The
use of single subject research to identify research-based practice in special education. Exceptional Children, 71.
Illinois Children’s Mental Health Partnership (2005). Strategic plan for building a
comprehensive children’s mental health system in Illinois. Retrieved March 1, 2006, from http://www.ivpa.org
161 Isava, D. (2006). An investigation of on the impact of a social emotional learning
curriculum on problem symptoms and knowledge gains among adolescents in a residential treatment center. Unpublished Doctoral Dissertation.
Jones, K.M., Wickstrom, K.F., & Friman, P.C. The effects of observational feedback on
treatment integrity in school-based behavioral observation. School Psychology Quarterly, 12, 316-326.
Joyce, B. & Showers, B. (1988). Student achievement through staff development.
Longman: New York. Kratochwill, T. R., Elliott, S. N., & Callan-Stoiber, K. (2002). Best practices in school-
based problem-solving consultation. In A. Thomas, & Grimes, J. (Ed.), Best practices in school psychology IV (pp. 503-516). Bethesda, MD: NASP.
Lewinsohn, P. M., Clarke, G. N., Seeley, J., R., & Rohde, P. (1994). Major depression in
community adolescents: Age of onset, episode duration, and time to recurrence. Journal of the American Academy of Child and Adolescent Psychiatry, 33, 809-818.
McEvoy, M.A., Shores, R.E., Wehby, H.H., Johnson, S.M., & Fox, J.J. (1990). Special
education teachers’ implementation procedures to promote social interaction among children in integrated setting. Education and Training in Mental Retardation, 25, 267-276.
Merrell, K. W., Blade, R. L., Lund, J., & Kempf, K. K. G. (2003). Convergent and
discrimination construct validity of the Internalizing Symptoms Scale for Children with the BASC-SRP-C. Psychology in the Schools, 40, 139-144.
Merrell, K. M., Carrizales, K., & Feuerborn, L., Gueldner, B. A., Tran, O. K. (in press).
Strong Kids-M: A social and emotional learning curriculum for students in grades 6-8. University of Oregon: Oregon Resiliency Project. Baltimore: Paul H. Brooks Publishing.
Merrell, K. M., Juskelis, M., & Tran, O. (2004, July). Evaluation of the strong kids and
Strong Kids curricula for promoting social and emotional resiliency. Presented at the meeting of the American Psychological Association, Honolulu, HI.
Merrell, K. W., Juskelis, M. P., Tran, O. K., & Buchanan, R. (2006). Social and
emotional learning in the classroom: Impact of Strong Kids and Strong Teens on students' social-emotional knowledge and symptoms. Manuscript submitted for publication.
162 Michael, K. D. & Crowley, S. L. (2002). How effective are treatments for child and
adolescent depression? A meta-analytic review. Clinical Psychology Review, 22, 247-269.
Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: Preparing people for
change. (2nd ed.). New York: Guilford Press. Miller, W. R., Sovereign, R. G., & Krege, B. (1988). Motivational interviewing with
problem drinkers:II. The drinker’s check-up as a preventative intervention. Behavioural Psychotherapy, 16, 251-268.
Moncher, F., & Prinz, R. (1991). Treatment fidelity in outcome studies. Clinical
Psychology Review, 11, 247-266. Mortenson, B.P. & Witt, J.C. (1998). The use of weekly performance feedback to
increase teacher implementation of a prereferral academic intervention. School Psychology Review, 27, 613-627.
New York State Office of Mental Health. Retrieved October 18, 2006 from
www.omh.state.ny.us/omhweb/new/PATAKI/ChMHACT06.htm Noell, G.H., Duhon, G.J., Gatti, S.L., & Connell, J.E. (2002). Consultation, follow-up,
and implementation of behavior management interventions in general education. School Psychology Review, 31, 217-234.
Noell, G.H., Gresham. F.M., & Gansle, K.A. (2002). Does treatment integrity matter? A
preliminary investigation of instructional implementation and mathematics performance. Journal of Behavioral Education, 11, 51-67.
Noell, G.H., Witt., J.C., Gilbertson, D.N., Ranier, D.D., & Freeland, J.T. (1997).
Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77-88.
Noell, G.H., Witt, J.C., LaFleur, L.H., Mortenson, B.P., Ranier, D.D., & LeVelle, J.
(2000). Increasing intervention implementation in general education following consultation: A comparison of two follow-strategies. Journal of Applied Behavior Analysis, 33, 271-284.
Noell, G.H., Witt, J.C., Slider, N.J., Connell, J.E., Gatti, S.L., Williams, K.L., Koenig,
J.L., & Resetar, J.L. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87-106.
163 Northup, J., Wacker, D.P;, Berg, W.K., Kelly, L., Sasso, G., & DeRaad, A. (1994).
The treatment of severe behavior problems in school settings using a technical assistance model. Journal of Applied Behavior Analysis, 27, 33-47.
Payton, J. W., Wardlaw, D. M., Graczyk, P. A., Bloodworth, M. R., Tompsett, C. J.,
Weissberg, R. P. (2000). Social and emotional learning: A framework for promoting mental health and reducing risk behaviors in children and youth. Journal of School Health, 70, 179-185.
Reinke, W. M. (2006). The classroom check-up: A brief intervention to reduce current
and future student problem behaviors through classroom teaching practices. Dissertation Abstracts International: Section B: The Sciences and Engineering, 66, 3935.
Schwartz, I. S. & Baer, D. M. (1991). Social validity assessments: Is current practice
state of the art? Applied Behavior Analysis, 24, 189-204. Sheridan, S. M., Welch, M., & Orme, S. F. (1996). Is consultation effective? A review of
outcome research. Remedial and Special Education, 17, 341-354. Smith, G. (2004). Smith bill to prevent youth suicide to pass senate. Retrieved September
30, 2006 from http://gsmith.senate.gov/public Stark, K. 1990. Childhood depression: School-based intervention. New York: Guilford
Press. Stokes, T. F. & Baer, D. M. (1977). An implicit technology of generalization. Journal of
Applied Behavior Analysis, 10, 349-367. Telzrow, C. F., Beebe, J. J. (2002). Best practices in facilitating intervention adherence
and integrity. In A. Thomas, & Grimes, J. (Ed.), Best practices in school psychology IV (pp. 503-516). Bethesda, MD: NASP.
Vermilyea, B.B., Barlow, D.H., & O’Brien, G.T. (1984). The importance of assessing
treatment integrity: An example in the anxiety disorders. Journal of Behavioral Assessment, 6, 1-11.
Wang, M. C., Haertel, G. D., Walberg, H. J. Fostering educational resilience in inner-city
schools. In H. J. Walber, O. Reyes, & R. P. Weissberg (Eds.), Children and youth: Interdisciplinary perspectives. Thousand Oaks, CA: Sage Publications, Inc.
Weisz, J.R., Donenberg, G.R., Weiss, B., & Han, S.S. (1995). Bridging the gap between
laboratory and clinic in child adolescent psychotherapy. Journal of Consulting and Clinical Psychology, 63, 688-701.
164 Wilcox, H. C. & Anthony, J. C. (2004). Child and adolescent clinical features as
forerunners of adult-onset major depressive disorder: Retrospective evidence from an epidemiological sample. Journal of Affective Disorders, 82, 9-20.
Wilson, D.B., Gottfredson, D.C., & Najaka, S.S. (2001). School-based prevention of
problem behaviors: A meta-analysis. Journal of Quantitative Criminology, 17, 247-272.
Witt, J.C., Noell, G., LaFleur, L., & Mortenson, B.P. (1997). Teacher use of intervention
in general education setting: Measurement and analysis of the independent variable. Journal of Applied Behavior Analysis, 30, 693-696.
Wolf, M. M. (1978). Social validity: The case for subjective measurement for subjective
measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203-214.
Zins, J.E., Bloodworth, M.R., Weissberg, R.P., & Walberg, H. J. (2004). The scientific
base linking social and emotional learning to school success. In J. E. Zins, R. P. Weissberg, M. C. Wang, & H. J. Walberg (Eds.), Building academic success on social and emotional learning: What does the research say? New York: Teachers College.
165
