Central Texas Freshwater Mussels Research Program, Meghan E. Hope
University of Texas Evaluation of Journey of Hope
Transcript of University of Texas Evaluation of Journey of Hope
1
THE JOURNEY OF HOPE CURRICULA:
BUILDING RESILIENCE AFTER A NATURAL DISASTER
Authors:
Lori Holleran Steiker PhD
Tara Powell MSW, MPH
The University of Texas
Evaluation of Save the Children’s
Journey of Hope Program in
Tuscaloosa, Alabama
2
TABLE OF CONTENTS
Executive Summary…………………………………………………………………………………... 3
Acronyms…………………………………………………………………………………………........ 6
1.0 Introduction…………………………………………………………………………………........ 7
2.0 Journey of Hope Program: Background and Overview
2.1 Background of Journey of Hope Program in Alabama……………………………………….. 9
2.2 Overview of the Journey of Hope Programs…………………………………………………... 9
3.0 Evaluation Objectives, Outcomes & Limitations
3.1 Objectives…………………………………………………………………………………........ 13
3.2 Limitations…………………………………………………………………………………….. 13
3.3 Intended Users…………………………………………………………………………………. 13
4.0 Evaluation Methodology, Scope and Results
4.1 Elementary and Adolescent Journey of Hope……………………………………………………... 14
4.1.1 Methodological Tools……………………………………………………………………....... 14
4.1.2 Setting………………………………………………………………………………………... 16
4.1.3 Data Analyses and Results…………………………………………………………............... 16
Elementary Journey of Hope
Written Questionnaires…………………………………………………………………… 16
Disciplinary Referrals……………………………………………………………………. 18
Youth Coping Inventory………………………………………………………………….. 19
Adolescent Journey of Hope
Communities that Care Survey…………………………………………………………... 21
Strengths and Difficulties Questionnaire………………………………………………… 21
Qualitative Findings Adolescent and Elementary………………………………………… 22
4.2 Teen Journey of Hope
4.2.1 Methodological Tools……………………………………………………………………........ 30
4.2.2 Setting………………………………………………………………………………………... 30
4.2.3 Analyses and Results…………………………………………………………………………. 30
5.0 Key Findings & Discussion
5.1 Elementary and Adolescent Journey of Hope…………………………………………………….. 34
5.2 Teen Journey of Hope……………………………………………………………………………... 36
5.3 Limitations………………………………………………………………………………………… 37
5.4 Conclusions………………………………………………………………………………………. 38
Appendices............................................................................................................................................. 40
Appendix 1: Teen Journey of Hope Questionnaire
Appendix 2: Strengths and Difficulties Questionnaire
Appendix 3: Written Questionnaires
Appendix 4: Qualitative Interview Guide
Appendix 5: CTC Survey
Appendix 6: Youth Coping Inventory
References............................................................................................................................................... 50
3
Executive Summary
Background
In response to a devastating tornado outbreak in Alabama in April 2011, Save the Children, in
partnership with University of Texas, conducted and evaluated the Journey of Hope (JoH)
program in Tuscaloosa, Alabama from September 2011-May 2012. The JoH program was
developed by Save the Children after Hurricane Katrina in response to the specific psychosocial
needs of children after that disaster. The premise of the program is to “help children and adults
cope, build on their natural resiliency and strengthen their network of social support with friends
and caring others.”1 Thus, the program’s goal is to increase children and adolescents’ knowledge
and skills to enhance coping strategies and protective factors that promote resilience. The JoH
program consists of four developmentally appropriate curricula that teach socio-emotional and
coping skills in an interactive learning environment facilitated by social workers and counselors.
Junior Journey of Hope
(JJOH)
Ages 5-7 Eight 30-45-minute sessions with groups of 5-7 children
focusing on topics such as safety anger, sadness, bullying, and
self-esteem incorporating cooperative play, art and literacy.
Elementary Journey of
Hope (EJOH)
Ages 8-10 Eight 1-hour sessions with groups of 6-10 youth focusing on
topics such as safety, self-esteem, anger and aggression
incorporating literacy, art, and music.
Adolescent Journey of
Hope (AJOH)
Ages 11-14 Eight 1-hr sessions with groups of 6-10 early adolescents
focusing topics such as stress, fear, safety, self-esteem and anger
incorporating games, journaling, and discussion.
Teen Journey of Hope
(TJOH)
Ages 15-18 Three 1-hr sessions with groups of 8-15 older teens and
emerging adults focusing on topics such as stress and coping
through games, discussion, and relaxation exercises.
Objectives
The 2011-12 Tuscaloosa JoH evaluation builds on previous findings from JoH evaluations
conducted in 2009 in New Orleans after Hurricane Katrina and in 2011 after a 6.3 magnitude
earthquake struck Christchurch, New Zealand. The New Orleans results were mostly qualitative
and found that after youth participated in the JoH curricula they expressed: increased social
well-being, emotional well-being and knowledge and skills. Quantitative and qualitative data
from the evaluation in New Zealand found youth who participated in the JoH exhibited reduced
emotional distress and increased emotional regulation and awareness. Building on previous
findings, the objectives of the Tuscaloosa evaluation were to:
Assess the impact of the Elementary and Adolescent JoH curricula in a post-disaster
setting
Inform continuous program improvement related to process, content, and evaluation
Methodology
The evaluators followed a mixed methods approach to the research, which utilized quantitative
and qualitative measures to assess the impact of the program on participants. The following
methods were used to evaluate each curricula:
1 Save the Children Fact Sheet
4
EJoH
Validated scales measured children’s coping a were delivered pre and post intervention
for both the experimental and control groups;
District level disciplinary referral data for children participating in the EJoH gathered in
the fall 2011 and spring 2012;
In-depth interviews and focus groups conducted post program participation measuring:
what topics resonated with participants, which topic was most important and what they
liked about the group;
A standardized survey measuring knowledge and skills gained from the program
delivered post intervention.
AJoH:
Validated scales measured children’s emotional distress, coping and school bonding were
delivered pre and post intervention;
In-depth interviews and focus groups conducted post program participation measuring:
what topics resonated with participants, which topic was most important and what they
liked about the group.
Key Findings
From a thorough analysis of all data, the results of the JoH program evaluation demonstrate
enhanced socio-emotional functioning in the following areas:
Decreased emotional distress
Reduced externalizing behaviors (i.e. school disruptions)
Enhanced interpersonal communication and peer relationships
Elementary Journey of Hope
Decreased emotional distress 1. Written standardized surveys indicated that youth who participated in the EJoH learned
how to make themselves feel better when having difficulties with emotions (mean of 4.73
on 5 point Likert scale); and
2. Thematic qualitative findings revealed that participants had increased abilities to deal
with stress and augmented coping skills gained from participation in the program.
Reduced externalizing behaviors
1. Disciplinary referrals dropped by 30% of those who participated in the Elementary
Journey of Hope program;
2. Written standardized surveys indicated youth learned about how to manage anger (4.83
on a 5.0 scale); and
3. Qualitative themes illustrated comments about understanding anger, recognizing the
related feelings, and new coping mechanisms.
5
Adolescent Journey of Hope
Decreased emotional distress 1. The Strengths and Difficulties Questionnaire (SDQ) illustrated that after adolescent boys
6th
-8th
grade and adolescent girls in 8th
grade participated in the program their emotional
distress symptoms were reduced from borderline emotional problems to a normal range.
Enhanced interpersonal communication and peer relationships
1. Students who participated in the AJoH self-reported improved peer relationships on the
Communities that Care Survey from pre to post-test (13.5 to 14.2); and
2. Qualitative data indicated youth enhanced friendships, trust, were able to talk with others
to feel better, and even understanding others (i.e., empathy).
Limitations
A major limitation was not being able to use a quasi-experimental study design to evaluate AJoH
due to an inadequate number of participants for a control group. Another limitation was not
being able to assess the impact of the program on participants beyond the end of the program
implementation period due to limited funding. Future studies would benefit from a longitudinal
study design.
Recommendations
Recommendation #1: Continue to implement the JoH program in response to disasters.
This is the second domestic evaluation of the JoH and indicates the curricula are positively
impacting their target audiences. Given the dearth of post-disaster, school-based psychosocial
programs, the JoH program provides a unique contribution to the field and fills a gap in
programming available for youth after a disaster.
Recommendation #2: Continue to evaluate JoH to ensure that it is an evidence-based program
which positively impacts participants.
While it is challenging to use a quasi-experimental study design in the midst of a disaster, the
timing of the curriculum evaluation is valuable to understanding its efficacy. Conducting
ongoing and consistent evaluations will increase the aggregate sample and thus the ability to
determine statistically significant changes resulting from the intervention. Ideally, in the next
evaluation, quantitative measures should be added for the following constructs which
thematically arose in qualitative findings including: self-efficacy, affect regulation, and social
skills.
Recommendation #3: Implement the JoH program with youth in settings other than disaster
response, for example, inner city youth who deal with stressors such as violence, crime and
poverty.
This will allow for more rigorous methodological design (being that the settings are not in a
period of crisis), and help determine the efficacy of the intervention without the inevitable
limitations of school settings post disaster.
6
Acronyms
Adolescent Journey of Hope AJoH
Communities that Care Survey CTC
Elementary Journey of Hope EJoH
Institutional Review Board IRB
International Institute of Children’s Rights and
Development
IICRD
Journey of Hope JoH
Junior Journey of Hope JJoH
Strengths and Difficulties Questionnaire SDQ
Teen Journey of Hope TJoH
Youth Coping Inventory YCI
7
1.0 Introduction
Save the Children Program in Alabama
During the week of April 25, 2011 a series of tornadoes struck the South, Midwest and
Northeastern United States. This tornado outbreak was the largest recorded in US history and
took hundreds of lives and left thousands displaced. Tuscaloosa, Alabama, was among the cities
devastated by those series of tornadoes. On April 27th
, 2011 an E-4 tornado, spanning more than
a mile wide struck Tuscaloosa, causing more than 1,000 injuries and approximately 65 deaths in
the area. This was the highest death toll from a tornado in the United States since 1955 (NASA,
2011).
Children were among those most adversely impacted by the tornadoes in Tuscaloosa. Hundreds
were injured or separated from their family, and many experienced loss of family members,
caretakers and peers. Moreover, these tornadoes not only displaced hundreds of children from
their homes, but also damaged or destroyed a number of schools.
Natural disasters such as hurricanes, earthquakes, tsunamis, floods and wildfires can impact a
child’s social and emotional well-being (Garrett et al., 2007; Kataoka, Rowan, & Hoagwood,
2009; Walsh, 2007). The psychosocial impact on children affected by disasters can be
debilitating and may appear immediately or surface weeks or months after experiencing the
event. The emotional strain on children affected by a traumatic event may be exhibited in a
variety of ways, including re-experiencing the event, intrusive thoughts, avoidance of similar
situations around the trauma, hyper-arousal, and anger (Wang et al., 2006). Externalizing
symptoms (anger and acting out at school and at home) and internalizing symptoms (anxiety and
depression) may also be mental health consequences for children who have experienced a
disaster (Jaycox, 2006). Furthermore, common post-disaster psychological disorders can include
acute stress reactions, adjustment disorders, depression, panic disorders, PTSD, and anxiety
disorders (Kar, 2009)
There are a variety of risk factors which can augment the likelihood of post-disaster
psychological symptoms in youth. Greater exposure to the disaster, witnessing others in life-
threatening situations, having family members die, demographic factors (age, gender), pre-
8
existing characteristics of the child, the post-disaster recovery environment, child’s
psychological resources, parental distress or length of displacement can put children at a higher
risk for developing post-traumatic stress symptoms (Cohen et al., 2009; Kar, 2009). Conversely,
research has shown protective factors for children such as parental and social support, promoting
a sense of control, normality, and empowerment may mitigate post-traumatic stress symptoms
(Cohen et al., 2009; Williams, Alexander, Bolsover, & Bakke, 2008).
The Journey of Hope curricula seek to mitigate the negative psychological sequelae after a
disaster by enhancing protective factors such as establishing a sense of control and normalcy
through discussion and activities, and providing psycho-educational knowledge about emotions
such as fear, anger, and anxiety that children may encounter after a traumatic event.
9
2.0 Journey of Hope Program: Background and Overview
2.1 Journey of Hope Programming in Tuscaloosa, AL
The Journey of Hope curricula were first brought to Tuscaloosa in September of 2011 because of
the unique psychosocial needs of the children who experienced the devastation of the tornado.
Hundreds of children were displaced from their homes and schools and had lost the sense of
stability they had before the tornado. As the city began to rebuild and schools were slowly
being re-opened, Save the Children partnered with Tuscaloosa city schools to respond to the
emotional needs of those children most affected by the disaster.
These developmentally appropriate set of curricula were provided to youth between the ages of 6
and 18 over the course of the 2011 and 2012 school year and to date have served 2,897 youth in
Tuscaloosa2.
Table 1: Beneficiaries of Journey of Hope Fall 2011
Adolescent Journey of Hope: 156
Elementary Journey of Hope: 467
Junior Journey of Hope: 476
Fall Total: 1,099
Spring 2012
Adolescent Journey of Hope: 295
Elementary Journey of Hope: 543
Junior Journey of Hope: 709
Journey of Hope for Teens: 231
Journey of Hope for Caregivers: 20
Spring Total: 1798
Overall Total: 2,897
2.2 Overview of Journey of Hope Programs
After a disaster, children continue to endure the everyday stresses of living in a community that
is rebuilding, often with limited access to services. Many of these children have witnessed
significant changes in their homes, neighborhoods, and schools and continue to process those
changes in their everyday lives. The Journey of Hope curricula were originally brought to
Tuscaloosa, Alabama, in response to the emotional strain on the youth who were impacted by the
tornadoes. As the schools were rebuilding and slowly re-opening, many of the youth were
exhibiting signs of distress. Numerous children were displaced or lost a loved one and all had
2 See Table one for breakdown of beneficiaries
10
experienced the devastation the tornado brought to the city. In response, Save the Children US
liaised with Tuscaloosa city schools to bring the Journey of Hope psychosocial curricula to
Tuscaloosa.
The Junior (JJoH), Elementary (EJoH), Adolescent (AJoH) and Teen Journey of Hope (TJoH)
are developmentally appropriate curricula that offer children between the ages of 6 and 18 the
opportunity to better normalize their emotions and develop positive coping strategies through
cooperative play, creative arts and literacy. These curricula, which are based in social cognitive
theory, teach children social and emotional skill building to promote self-efficacy, problem
solving and positive coping so they may have the capacity to overcome current and future
traumas (Bandura, 1998). Additionally, the Journey of Hope curricula enhance protective factors
for youth such as promoting healthy peer relationships, a positive school environment, and stable
relationships with adults.
The JoH curricula, which have been administered in Tuscaloosa city schools since September,
2011, are designed to teach children the skills to overcome the adversity of a natural disaster.
The premise of the curricula is: “to help children cope, build on their natural resiliency and
strengthen their network of social support with friends and caring others”.3
Working in coordination Save the Children U.S., consultants from the University of Texas
embarked on a comprehensive evaluation to identify the impact of the Journey of Hope curricula.
These findings build on previous findings from an evaluation conducted in 2009 by the Institute
for Child Rights and Development (IICRD) in New Orleans after Hurricane Katrina and from a
recent evaluation conducted in Christchurch, New Zealand, in 2011 in response to a 6.3
magnitude earthquake that struck the city. Following are specific learning objectives of the
Journey of Hope curricula.
Learning Objectives
1. To support children in understanding and normalizing emotions associated with trauma or
difficult circumstances;
2. To support children in developing positive coping strategies to deal with these emotions;
3 Save the Children Fact Sheet
11
3. To build on the innate strengths of children, their families, schools, and communities to further
develop positive coping mechanisms; and
4. To instill a sense of hope, empowering children to feel more in control over stressors.
Program Design
The Journey of Hope curricula use a child-centered, strengths-based approach to provide
children, adolescents and emerging adults with positive resources to understand and cope with
emotions caused by traumatic situations. The curricula are organized into 30 minute to hour-long
sessions that can be implemented within a school term or in a summer camp4. The core tenets of
the JoH are to help children, adolescents, older teens and emerging adults: 1) understand and
normalize key emotions; 2) identify triggers and stressors; and 3) develop positive coping
strategies to deal with these emotions. Table 2 provides a description of each program including
length, topics and structure.
Table 2: Description of JoH programs
Program Description
Junior Journey of Hope (ages 5-7) Eight 30-45-minute sessions with groups of 5-7 children focusing
on topics such as safety, anger, sadness, bullying, and self-esteem
incorporating cooperative play, art and literacy.
Elementary Journey of Hope (ages 8-10) Eight 1-hour sessions with groups of 6-10 youth focusing on topics
such as safety, fear, anger and aggression and self-esteem
incorporating games, books, art, and music.
Adolescent Journey of Hope (11-14) Eight 1-hr sessions with groups of 6-10 early adolescents focusing
topics such as stress, fear, safety, self-esteem and anger
incorporating games, journaling, and discussion.
Teen Journey of Hope (15-18) Three 1-hr sessions with groups of 8-15 older teens and emerging
adults focusing on topics such as stress and coping through games,
discussion, and relaxation exercises.
Each session of the Journey of Hope curricula follows a similar routine to create a safe place
where participants feel comfortable participating in activities and sharing their feelings to help
normalize emotions. Moreover, each program utilizes developmentally appropriate learning
strategies, including:
Cooperative games to enhance social skills, encourage teamwork, and build awareness of
stressors in a non-competitive manner;
Books and dialogue to enhance emotional intelligence and reinforce messages to help
normalize emotions after a trauma; and
4 See Table 2 for a description of each Journey of Hope program
12
Music, art, journaling and dance and/or movement to give children an opportunity
process their emotions through an alternative outlet of expression.
Each program is adapted to use age appropriate activities. The core content of the JJoH, EJoH
and AJoH are as follows:
The core content and structure of the TJoH is as follows:
Session Topic:
1 Introduction: Creating Safety
2 Fear: Understanding and Coping
3 Anxiety: Understanding and Coping
4 Sadness: Understanding and Coping
5 Anger and Aggression: Understanding and Coping
6 Bullying: Understanding and Coping
7 Self-Esteem and Taking Action: I Believe I can
8 Me, My Emotions and My Community
Session Topic:
1 Introduction: Types and Sources of Stress
2 Stress Reactions: Understanding and Coping
3 Coping Mechanisms: Resources and Coping
13
3.0 Evaluation Objectives, Anticipated Outcomes and Limitations
This section of the evaluation describes the objectives, outcomes, and limitations of the
evaluation process including: (1) core objectives of the evaluation; (2) limitations; and (3) the
intended end users of the evaluation.
3.1 Objectives of the Evaluation
The key objectives of the evaluation were to:
Objective 1: To assess the impact of the Elementary, Adolescent and Teen Journey of Hope
curricula in Tuscaloosa as delivered by social workers and counselors;
Objective 2: To evaluate the applicability of the JoH curricula in a post-disaster
context; and
Objective 3: To provide Save the Children with results from the evaluation, aimed at delivering
a more thorough knowledge on the curricula’s impact.
In accordance with the objectives, the anticipated outcomes of the evaluation were to:
To provide a more thorough evidence base of the Journey of Hope curricula for Save the
Children;
To deliver a solid understanding of the applicability of the JoH curricula (Elementary,
Adolescent and Teen); and
To provide information on improvements that could enhance the impact of the curricula.
3.2 Limitations of the Evaluation
The following limitations are noted:
The absence of a control group for the Adolescent and Teen Journey of Hope was a
limitation of the study due to missing consent forms;
Missing data made it was impossible for analysis of the Strengths and Difficulties (SDQ)
for youth who participated in the EJoH; and
Because of the short duration of the evaluation funding period, long-term follow-up was
not feasible. Future studies would benefit from a longitudinal study design.
3.3 Intended end users of the evaluation
Following is list of recognized stakeholders who may be end users of the evaluation:
Save the Children staff (psychologists, social workers and the management team);
Save the Children Alliance personnel and management;
Communities impacted by a disaster both in Tuscaloosa and globally; and
Agencies and schools within Tuscaloosa who have received the program.
14
4.0 Evaluation: Methodology, Scope and Results
4.1 Elementary and Adolescent Journey of Hope
The Elementary and Adolescent Journey of Hope curricula were first delivered in September,
2011. To date the curricula have been implemented with 1460 youth in Tuscaloosa city schools.
The evaluation used methodological tools which will be described below. Due to the
developmental age of the youth and sensitivity of survey instruments a variety of scales were
distributed based on the age of the participants. The sample includes the following:
The Youth Coping Inventory was completed at pre and post-test by N=61 (waitlist-
control group) and N=73 (experimental group) who were between the ages of 7-10 who
participated in the EJoH and with N=106 students 11-14 who participated in the AJoH;
The Communities that Care questionnaire was completed with (N=106) adolescents that
participated in the AJoH;
Disciplinary referrals were gathered pre and post intervention for all students who
participated in the evaluation;
The Strengths and Difficulties Questionnaire was completed by (N=106) students who
participated in the AJoH;
Save the Children’s written questionnaires were completed by N=71 youth who
participated in the EJoH; and
Individual interviews were conducted with N=30 students after participation in the EJoH
and AJoH curricula.
4.1.1 Methodological Tools
To attain a comprehensive understanding of the impact of the curricula, specific measures were
used. The evaluators used both quantitative and qualitative methodological tools. Validated
scales that measured coping skills, school bonding, and emotional symptoms of the participants
were completed both before and after participation in the JoH program. Written questionnaires
were also distributed post-program to measure knowledge and skills gained through
participation, and qualitative interviews were conducted using open-ended questions about what
the participants gained from the program.
15
The following data collection methodologies were used:
The SDQ The Strengths and Difficulties Questionnaire (SDQ) is an evidence-based
brief 25 item scale which is a behavioral screening questionnaire for
children (parent and teacher report ages 3-10) and adolescents (self-
report ages 11-16) (Goodman, 2001). The SDQ consists of five subscales
measuring emotional symptoms, conduct problems,
inattention/hyperactivity, peer relationship problems, and pro-social
behavior. The scale has high internal reliability using Cronbach’s Alpha
of .73. The internal consistency of SDQ has also been supported across
multi-cultural settings including; Dutch (.80), UK (.88), and Australia
(.77). An analysis of SDQ subscales in Australia also illustrated moderate
to strong internal reliability (Hawes & Dadds, 2004).5
Written A scale created by Save the Children staff in New Orleans and the
Questionnaires International Institute of Children’s Rights (IICRD) was used to measure
knowledge and skills the children gained from participating in the EJoH
program. This five point Likert scale, measured participants’: (1) favorite
activities; (2) knowledge about anger and aggression; (3) comfort in the
group; (4) feelings of safety in the group; and (5) knowledge about the
impact of bullying.6
Interviews Qualitative interviews were conducted using an interview guide for the
EJoH and AJoH. This guide asked questions such as: (1) what
participants liked about the group; (2) which feeling was most important;
(3) what participants didn’t like about the group; and (4) the favorite
activities of the participants.7 The guide was developed by IICRD during
the pilot evaluation in New Orleans. It was then adapted to fit the needs
of the participants in Tuscaloosa.
Disciplinary The number of disciplinary referrals was obtained from Tuscaloosa City
Referrals Schools8. The disciplinary referrals were gathered at two time points, the
term prior to participation in the Journey of Hope program and the term
following participation in the program.
5 See appendix 2 for Strengths and Difficulties Questionnaire
6 See appendix 3 & 4 for Written Questionnaire
7 See appendix 5for qualitative interview guide
8 Disciplinary referrals included: excessive distraction of other students, fighting, vandalism, offensive touching,
threat, harassment or intimidation of student or other offenses deemed inappropriate by the principal.
16
Community
that Cares
Survey
The Community that Cares (CTC) survey is a validated measure that
assesses risk and protective factors for adolescent problem behaviors
such as delinquency and substance use. The CTC has a series of sub-
scales that measure 10 protective factors and 22 risk factors. For the
purpose of the Journey of Hope evaluation we used the school bonding
sub- scale.9
School Bonding
The school bonding sub-scale measures protective factors such as liking
school, time spent on homework, and perceiving schoolwork as relevant.
This subscale is also negatively related to drug use and other outcomes
such as dropping out of school. Internal consistency reliability for the
School bonding subscale ranged from .71-.77(Arthur, Hawkins, Pollard,
Catalano & Baglioni, 2002).
Youth Coping The YCI is a self-report measure of coping style. Participants rate the
Inventory(YCI) frequency of their use of various coping strategies on 31 items,
scored on a 5-point Likert scale. Responses are categorized into three
broad coping strategies. Internal consistency for the YCI is high
(Cronbach’s alpha = .86) (McCubbin et al., 1996). The predictive validity
of the YCI has also been established (McCubbin et al., 1996).
4.1.2 Setting
The evaluation followed ethical Institutional Review Board (IRB) guidelines established by the
University of Texas at Austin. Each child who participated in the pre and post-test test
questionnaires and qualitative interviews was provided permission by their parent or guardian
through a consent form and assented to participate. The school principals and social workers
liaised with Save the Children psychologists and social workers to gather the consent forms from
the parents.
4.1.3 Data Analysis and Results
Elementary Journey of Hope
Written Questionnaires
Written questionnaires were completed post-intervention to assess each component of the
intervention including: favorite activities, knowledge about anger and aggression, comfort in the
9 See appendix 6 for CTC survey
17
group, feelings of safety in the group and knowledge about the impact of bullying.10
Descriptive
statistics were conducted to find the means of the survey items. Following are results of the
written questionnaires for the Elementary Journey of Hope intervention. The sample of written
surveys for the EJoH program included (N=71) students between the ages of 6-12. The majority
of respondents were female (N=42, 59.2%) and were a mean age of 9 years old.11
The post-test questionnaire was completed on a five point Likert scale with a score of 1
indicating “not at all” and a score of 5 equaling “a lot”. Findings yielded positive results with
students overwhelmingly (4.49 to 4.83 on the Likert scale) expressing that they enjoyed the
group, learned specific knowledge and skills and liked the activities.12
Notable findings from the
written surveys include that group participants rated: learned how to make themselves feel better
(4.73); learned about bullying; (4.82) and learned how to handle anger (4.83)-the highest on the 5
point scale.
Table 3: Demographic information
Age N Percent
6 1 1.4
7 9 12.7
8 11 15.5
9 29 40.8
10 12 16.9
11 5 7.0
12 1 1.4
Missing 2 2.8
Total 71 100.0
Gender N Percent
Female 42 59.2
Male 27 38.0
10
See appendix 3 for complete written questionnaire 11
Table 3 provides demographic information 12
Table 4 illustrates the mean scores for each of the questions from the written survey
18
Table 4 provides the mean scores for the written questionnaires:
Table 4: Means of Written Questionnaires for the EJoH
Questions Mean
Shared Feeling 4.61
Like Activities 4.80
Respected in group 4.42
Rules 4.49
Felt Safe in group 4.41
Wanted to go to group 4.62
Learned about bullying 4.82
Learned how to handle anger 4.83
Like the Books 4.57
Learn about self 4.53
Learned to make self feel better 4.73
Disciplinary Referrals
District level disciplinary referral data was gathered for all students who participated in the
Elementary Journey of Hope program and had parental consent. The sample of disciplinary
referral data included (N=126) students who participated in the EJoH. Findings indicate a
decline in disciplinary referrals between baseline and post participation in the EJoH. At time one
(December, 2011), there were 36 reported disciplinary referrals of the students who participated
in the EJoH and time two (May, 2012) there were 25 referrals. These findings are promising
being that one sign of distress among youth is externalizing behaviors. Furthermore, as will be
outlined in the qualitative section, the ability to manage anger and aggression were specific
themes that arose. Additionally, the highest mean on the written surveys was that students
learned how to manage anger (4.83) which further supports that the Elementary JoH program
may have an impact on externalizing behaviors.13
13
Table 5 provides a graph of the change in disciplinary referrals from the fall of 2011 and spring of 2012
19
Table 5: Disciplinary Referrals EJOH
Youth Coping Inventory
Quantitative data for the Elementary Journey of Hope used a quasi-experimental pre and post-
test design. The sample in the experimental group consisted of (N=73) youth between the ages of
7-12 who participated in the Elementary Journey of Hope program and (N=61) youth who were
in the waitlist-control group. The data were collected in four schools (Alberta, Central,
University Place Elementary, and Arcadia), and were evenly split between grade and gender.
Demographically the majority of participants were African American (74.9%), followed by
Caucasian (11.6%) and Latino (5.5%).14
Findings indicate a positive change in means in the experimental group for the communication
and tension management sub-scale of the YCI, whereas the wait-list control group means slightly
increased.15
While the changes were not statistically significant, there were differences in the
means between experimental and control group. Furthermore, the experimental group’s change
was nearing significance and was in the direction predicted by the hypothesis that participation in
the EJoH may have a positive impact on externalizing behaviors as measured by the
communication and tension management subscale of the YCI.16
Future studies would benefit
from adding validated teacher or parent report measures in addition to more self-report scales
14
See table 6 for complete demographic information 15
Lower scores indicated a higher ability to manage anger. Items 18, 23 and 25 were recoded 16
See table 7 for mean changes in the YCI
0
5
10
15
20
25
30
35
40
Term 1 2012 Term 2 2012
Disciplinary Referrals EJOH
Disciplinary Referrals EJOH
20
which would provide a more comprehensive picture on the impact of the EJoH on externalizing
behaviors.
Table 6: Demographics of Elementary Journey of Hope participants who completed the YCI
Grade %
3rd
38.5%
4th
34.9%
5th
26.5%
Gender
Female 50.2%
Male 49.8%
Race
African American 74.9%
Asian .7%
White 11.6%
Hispanic 15.5%
Table 7: Mean differences YCI communication and tension management
Mean Pre test Mean Post-test N Significance
Experimental group 2.44 2.20 73 .075
Control group 2.41 2.43 61 .823
Adolescent Journey of Hope
Quantitative data for the AJoH used a pre, post-test design with (N=105) students who
participated in program. A quasi-experimental design was not completed because of the
difficulty identifying students who were going to participate in the following round of
implementations, which resulted in an inability to obtain parental consent. The sample of the
AJoH included students between 6th
and 8th
grade, were mostly female (79%) and the majority
were African American (79.8%) followed by Hispanic (11.3%) and Caucasian (8.1%).17
17
See table 8 for demographic information
21
Table 8: AJoH demographic information
Grade %
6th
44.4%
7th
39.5%
8th
16.1%
Gender
Female 79.0%
Male 21.0%
Race
African American 79.8%
Asian .8%
White 8.1%
Hispanic 11.3%
CTC-school bonding
Paired sample T-tests indicated students who participated in the AJoH raised their school-
bonding scores by almost a point between pre and post-test. This finding is particularly
noteworthy because peer bonding is considered a protective factor against future negative
psychological outcomes.
Table 9: Means differences CTC school bonding pre and post-test
Mean Pre test Mean Post-test N Significance
Adolescent JOH 13.5 14.2 105 .07
Strengths and Difficulties Questionnaire
The SDQ illustrated gender specific outcomes including positive changes in total emotional
symptoms and reduced peer problems among boys, and reduced conduct problems for girls.
Table 10 indicates that peer problems among males were reduced from 3.28 to 2.60. This is
particularly notable because a score above 3.0 falls in the borderline range of peer difficulties,
while below 3.0 is in the normal category for peer problems. Similarly, the total emotional
symptoms for boys were reduced from 11.84 (borderline) to 10.92 (normal). The mean
difference of conduct problems for girls was also noteworthy because their self-reported
outcomes illustrated a drop in symptoms of .5. Moreover, the changes in all measures can be
identified as clinically significant because of the fall from borderline range to the normal range
of functioning (Ogles, Lunnen, & Bonesteel, 2001). According to Ogles, Lunnen & Bonesteel
22
(2001) clinical significance can indicate a meaningful change when a person is “empirically
indistinguishable” from peers who are “normal”, after participating in a mental health or
psychosocial program.
Table 10: Mean differences Peer Problems SDQ (Boys only) Mean Pre test Mean Post-test N Significance
SDQ Peer problems 3.28 2.60 25 .06
Total Emotional
Symptoms
11.84 10.92 25 .331
Table 11: Mean differences Conduct Problems SDQ (Girls Only)
Mean Pre test Mean Post-test N Significance
Conduct Problems 2.88 2.57 91 .06
Other noteworthy quantitative findings were the differences of 8th
graders in relation to 6th
and
7th
graders. The 8th
graders reported reduced conduct problems and reduced emotional
symptoms on the SDQ and increased problems solving skills on the YCI. Conduct symptoms on
the SDQ significantly dropped from 2.55 to 1.75 (p<.05) and problem solving skills on the YCI
significantly increased from 33.12 to 36.24 (p<.05). Emotional symptoms also fell from 12.30
(borderline emotional symptoms) to 10.90 (normal emotional symptoms). Moreover, the change
in means can again be identified as clinically significant considering the scores fell from the
borderline range of emotional symptoms to a normal level of functioning.
Table 12: Mean differences 8th
Grade only
Mean Pre test Mean Post-test N Significance
SDQ conduct 2.55 1.75 20 .01
YCI Problem Solving 33.12 36.24 20 .02
Total Emotional
Symptoms
12.30 10.90 20 .241
Qualitative Findings
The last week of January, the researchers conducted interviews and focus groups in three schools
(University Place, Eastwood, Alberta) where the Elementary and Adolescent Journey of Hope
were implemented. One interview guide was used for the EJoH and AJoH given the similarities
in the program design and structure. The evaluative inquiries revolved around questions about
23
the following: (1) what the students learned in the group, (2) what they liked about the group,
and (3) what they didn’t like or thought could be improved. The primary analysis method
utilized was the semi-structured interviews, or guided conversations with the goal of discovering
the informants’ experiences (Glesne & Peshkin, 1992). The interview schedule was semi-
structured, meaning that it had basic questions to start with, but those were followed up with
intuitive responses to the participants’ answers. This allowed for the emergence of the
participant’s agenda rather than the researcher’s sense of what is important. The interview
schedule was written by the research team based on their experience and knowledge of the
Journey of Hope curricula and the literature on children/adolescents, trauma, loss, and coping.
Most of the questions and probes were open-ended to elicit the participants’ beliefs, thoughts,
and experiences in their own words. The interviewers were particularly careful not to use labels
and descriptors that might bias the responses.
Interview guidelines were utilized to frame the narrative-style interviews. We began with a brief
explanation that we are asking them what they thought of the program. As the interviews
unfolded, the researchers’ goal was to obtain a balance between inviting the participants toward
the general areas of the research questions (e.g., experience of the Journey of Hope program,
impact of the lessons, feelings about their experiences and the effects of the program in relation
to their experience) and allowing them to feel free to discuss whatever was important to them.
The protocol for group formation emerged from the context--the participants’ administrators,
teachers, and counselors chose participants and formed the groups to be interviewed. With the
prerequisite that the student had parental/guardian consent and participated in at least six of the
eight sessions in Journey of Hope program, students were referred for interview by the teachers
and counselors. For the elementary school interviews, both interviewers sat in each interview
and focus group to allow one to focus primarily on recording data and the other to facilitate
process. This also allowed for one to escort the students to and from the classrooms while the
other prepared the space. The middle school interviews were done by one interviewer to ensure
that the student felt comfortable in the one-to-one setting and so that they did not feel a heavy
adult presence which might inhibit their responses. Each interview lasted approximately 30-45
minutes depending upon the number of students (interviews were shorter in duration for dyads
24
and individuals than for larger focus groups). The opening question addressed the student’s
perceptions of the Journey of Hope program activities and the rest of the session was spent
eliciting elaborations on the students’ responses. The role of primary facilitator was alternated,
while the other researcher observed and recorded data. Audiotapes accompanied the researchers’
written notes.
Data Analysis
The interviews were transcribed by two scholarship winning social work undergraduate students
who were both familiar with the curriculum, the project, the researchers and the present research
protocol. There were a total of 6 interviews and 8 focus groups. The analysis of the transcribed
data involved the process of coding to elicit patterns and themes in the data. The coding
involved breaking codes into subcategories reflecting the participants’ conditions, interactions,
strategies, consequences, styles while moving to increased specificity (Lofland & Lofland, 1995;
Strauss, 1987) Ultimately, themes were identified by ideas that occurred repetitiously.
N-Vivo software was utilized as well as traditional manual coding when analyzing the data. The
N-Vivo program, used to aid in the organization and analysis of the data, involves the coding of
the data in “tree structures” at increasingly integrative levels. It also allows for specific word
searches, juxtapositions, and frequency of words or phrases. The combination of computer and
traditional manual coding allowed for systematic and efficient analysis as well as time to reflect
and think about the connections and themes. Even when using the computer as a means for
analyzing qualitative data, the process is both creative and mechanical. Richards and Richards
(1994) make the distinction between “textual level operations” (e.g., moving of the data) which
are done by the computer, such as retrieving codes, and “conceptual level operations”
(development of themes) done by the person. Ultimately, the researcher builds relations between
the data and the themes.
The qualitative analysis began with 14 broad initial codes which included: games, feelings,
books, natural disasters, drawing, bullying, safety, learning, anger, friends, trust, peer groups,
learning about self, and sadness. The researchers then conducted more focused coding. The N-
Vivo code tree was used to make the broad codes more specific. Next, codes were evaluated to
see which were used more than others, less productive codes were omitted, and the most
25
resonant ones were selected. Codes were collapsed, supported or dropped. Ultimately, the
coding procedure proceeded until core categories emerged to the point of saturation (i.e., where
further analysis does not elicit new themes). The final and most noteworthy themes revolved
around the concepts of feeling regulation, dealing with anger/bullying, valuing the program,
self-efficacy, and finding new ways to cope.
The students described on numerous occasions how the group helped them to realize things that
people have choice in how they deal with their feelings and that when they aren’t working, they
can find new ways to deal with the feelings. For example, a student notes:
“Me, like I’ll have really (bad) feelings, that I could…I used to before the group, but
when the group happened I learned how to cope with some of it, how was a good way
to deal with something. What I liked is that, it’s . . . it’s not a place where you like have
to learn. You learn stuff but you also have fun while you learning and it’s also good to
help people who get out of control with their anger like me and I’m, and it helped me
learn how to control it more better and that’s how, that’s why I like it.”
And another student adds:
“Because, because like, it really helps with your feelings and everything. I mean when-
before I started this program I was really, really mad all the time, and now I’m just
like, not that mad anymore and stuff. And I really think it would help somebody with
anger issues and stuff like that.”
One student even recognized that the feelings could even be dangerous if positive coping
mechanisms were not utilized:
“Because like some people I know, they get angry a lot, so they, when they talk about
anger it could help them. Some people are going through lots of different things, like
they’re going through grief so that could help them, and also, they tell us about ways
you can like let your anger out without going to the extreme, ‘cause, like the extreme
would be, um, you trying to hurt yourself or trying to hurt others because you’re-
you’re so angry or you’re so upset, that’s how you let it out. And some people like,
she’s said, don’t go to the extremes, like when you were very upset, and stuff, you
think, “Oh I just want to leave the planet,” and you try to hurt yourself. Well, um, she
just said don’t do that because you have to think about how would your family feel,
how would your friends feel, and, uh, stuff like that.”
26
Several students describe how violence or bullying is a reaction that can be changed through new
awareness and choices:
“Wow, uh, you know, how, you know, this whole school basically is just…these kids,
some of them are, you know, don’t have that coping skill……they just react at either a
fight or altercation, but since…….we’ve been going to the program, and, uh, it’s kind
of fun because, you know, going in there, and since we—since we’re done, this year,
um, we, since then all we’ve, uh, you know, covered how to cope and today in class this
girl named H., she um, she was bothering me so, you know, I said something back of
course, and then, um, I actually stopped myself before I said what I really wanted to
say, so, I was like, ‘You know what? Cope.’”
Students not only learned how to deal with their own anger, but also how to empathize with
others who are angry and might bully as a coping mechanism.
As one student articulated:
“Uh, I learned that there are some, like, we don’t know what’s going on with them.
Like not saying that I did, but like sometimes bullies should think that, ‘cause they
don’t know what they’re going through. But I know I didn’t bully anybody in that
group. It’s just the fact that, um, well, some people they were going through stuff and
like when people bullied them or tried to make them feel bad, it didn’t help them at all.
It just made them feel worse. So I learned in the group—how other people were…I saw
some people have problems going on at home, some have problems that they are going
on in their heads, and we don’t know anything about them, so I learned that, I think
that bullies should just stop and think for a second, like what they’re going through
and stuff.”
One student even expressed increased positive sense of self from the changes experienced from
the program: “I was proud to be in the group ‘cause I learned how to control my anger.”
While many of the students commented on how much “fun” they had throughout the program,
especially noting the games (bullying game, parachute, fleece ball game, body guard, dancing,
journal, and parties among the favorites), the students readily articulated the value of the games
in augmenting their self-awareness, affect regulation abilities, and coping skills. For example,
a student stated:
“And the games helped us with a skill. You know to have fun but at the same time help
us understand. Like one day I came in and I was angry about something and then we
27
played this little toss around the ball game and then, you know, it kinda made me feel
happier.”
Another child understood the implications of the affect regulation feeling work on a deeper level,
stating the following:
“How you feel sometimes, how you feel like you can let it, to let it out sometimes you
feel better. So we talked about different ways you could like, um, in your notes—like
you could have a diary or you could just talk it out to yourself or stuff like that, or you
could call a friend and talk to them. And then on anger, well, um, it helped you learn
how to, um, like just calm yourself down every now and then. How to relax. ‘Cause
sometimes I know I get angry, but, um, like it helped me to just think about what I
learned and stuff.”
Many of these comments reflect the theme of self-efficacy – the program clearly enhances
students perception that they are able to handle themselves better than before the program and be
good to themselves and others. This enhances their sense of safety, comfort, as well as their
social abilities. As stated by one student:
“All I know is that I had fun, and then, like, I remember before I even started this
program I was always like mad and getting on people’s nerves and now I do that less
and I’m like happy now.”
It emerged from the data that the students felt safe in the groups to talk about what they needed
to and to express themselves openly. For example, a student expressed:
“I could tell I could trust the group, and—‘cause like I said at the beginning, like
nothing ever went out of the group and, you know, I was just waiting for that one day
someone’d come up to me, ‘So this really happened?’ and like something like that, but
that never happened.”
This creation of a safe “holding environment” allowed for the exploration and learning that is
evidenced by their descriptions of their new coping skills and comfort levels. The importance of
trust is expressed throughout the interviews.
As one student states:
28
“We could trust them (the facilitators) a lot and I like to know that I’m able to trust
people because if you say that I can trust you but then you say you can’t, that I can’t-
that I can’t trust you, I’m probably gonna get very furious and just tell you just to leave
me alone because you’re playing with me and I don’t like to be played with about
certain feelings and emotions.”
As described by one student:
“Okay the thing I liked was, um….you know, you can go in there every day knowing
that, she said everything in that room stays in that room, confidential between the
people in that session and Miss K and Miss D. And, um, you know, you can go in there
and you can tell them everything, you know, you can tell them about what your day
was like….what your future is, and they won’t tell anything. They’ll try to explain to
you or plot it out how-what you should do for the problem that you have, and then, you
know, they just know what to say.”
And another student, commenting on a male facilitator:
“He asked us if we understood. And like took, and he asked us “Do we have anything
we can talk about?” And we can talk about everything with him. And he, uh, always
have an answer for us
29
Table 13: Summary of Salient Qualitative Themes
Questions
N
%
What did you do in the group?
Played Games 3 10.0
Talked about feelings 12 40.0
Talked about bullying 10 33.3
Learned about who or what feels safe 4 13.3
Learned about how to deal with anger 4 13.3
What did you learn?
How to deal with sadness 5 16.6
That it is normal to be scared 4 13.3
Not to take anger out on others 4 13.3
How to deal with bullies 6 20.0
What was your favorite activity?
Parachute 3 10.0
Discussion of what others liked about group members 3 10.0
Bullying game 5 16.6
Body guard 4 13.3
Team Juggle 2 6.6
Which feeling was most important? Bullying 7 23.3
Happiness 2 6.6
Fear 2 6.6
Anger and Aggression 9 30.0
Safety 1 3.3
Self-esteem 3 10.0
Sad 1 3.3
30
4.2 Teen JoH
The Teen Journey of Hope (TJoH) was first implemented in Tuscaloosa in the spring of 2012.
The program was adapted from the Caregivers Journey of Hope as a three session model to
identify stressors, help build coping skills and build supports for the teens. The evaluation in
Tuscaloosa was the first to be conducted for the TJoH, and was completed with nine groups
(N=139) of high school students who participated in the program in the spring of 2012. The
study used a pre/post-test design to identify the knowledge and skills gained from the program.
The researchers were then able to conduct statistical analyses to evaluate what the participants
gained from the curriculum.
4.2.1 Methodological Tools
To assess the outcomes of the Teen Journey of Hope, the evaluator adapted a previously used
quantitative survey that was used for the Caregiver’s Journey of Hope. The design included a
pre and post-test design which evaluated the knowledge and skills gained through the Teen
Journey of Hope program with (N=139) teens who participated in the intervention. Participants
who were evaluated filled out a survey that asked questions about knowledge before taking part
in the program and after completion. The survey instrument was based on a Likert scale ranging
from 1 (not at all) to 5 (A lot). Questions revolved around the following areas: (1) knowledge
about stress, (2) identifying personal and community supports, and (3) understanding coping
strategies.18
4.2.2 Setting
The setting that was used in the evaluation included three city schools where the program was
delivered. To be eligible to complete the Teen Journey of Hope questionnaire, the participants
were required to complete a registration form and attend the three session program.
Demographically, the participants were 47.8% (N=66) female and 49.7% (N=71) male, were
evenly split between grades 9-12 and 93.7% (N=134) identified as African-American19
.
4.2.3 Data Analysis and Results
Descriptive statistics were conducted using the statistical software (SPSS) to identify the
demographic characteristics of the sample and whether the program met participants’
18
See appendix 1 for the Teen Journey of Hope Survey instrument 19
See table 14 for Demographic Information
31
expectations and appropriate comfort level. Paired samples T-tests were then completed to
indicate whether there were statistically different results from baseline to post-test.
Table 14: Demographic Information Table 15: Comfort and Expectations
N %
Grade
9 28 21.2
10 39 29.5
11 36 25.2
12 29 22.0
Female 66 47.8
Male 71 49.7
The outcomes of the paired t-tests yielded significant (p<.05) improvements on four of the
survey items: (1) signs of stress, (2) coping, (3) how to manage stress and (4) types of stress.
The most significant changes in means were knowledge about types of stress (2.91 pre-test, 3.86
post-test) and knowledge on coping strategies to handle stress (3.23 pre-test, 3.86 post-test).
Other positive changes in means, although not significant, showed an increased awareness of
breathing exercises to help reduce stress, knowledge about how stress affects the body and
identifying social supports as a way to improve well-being20
.
Descriptive statistics also measured the level of comfort participants felt and whether the
program met their expectations. Results indicate 73.1% (N=84) participants rated a four or five
on whether the program met their expectations, and 65.9% (N=83) scored a 4 or 5 on comfort
level while participating in the curriculum21
.
The participants were also provided two open ended questions about: (1) what they liked about
the program and (2) ways the program could be improved. A thematic finding of the participants
who completed the open ended questions reported they enjoyed the activities and the games22
.
One response about the activities was:
20
See table 16 for mean scores from the TJoH quantitative survey 21
See table 15 for complete scores on comfort and expectations 22
27 participants stated they enjoyed participating in the games and activities.
N %
Expectations N %
3 (Somewhat) 22 19.1%
4 31 27.0%
5 (A lot) 53 46.1%
Comfort
1 (not at all) 8 6.9%
2 10 8.6%
3 (Somewhat) 15 12.9%
4 35 24.5%
5 (A lot) 48 41.4%
32
“I liked the games most, and that I was able to feel comfortable and able to express
myself.”
Another theme from the open ended responses was that participants found it helpful to learn
about how to cope with stress23
. Specific responses from participants included:
“JOH is an excellent program that is great to help people find positive ways to cope
with stress.”
and
“How we can talk about everybody's stress and how to deal with the different types of
stress, how to cope and what we do to relieve our stress.”
While almost half of the participants (N=26 of 57 responses) stated they enjoyed the program as
it was, suggestions on how to improve the program included: (1) the TJoH needs more sessions
to further discuss stressors and coping, and (2) the sessions needed to be longer because of the
valuable information that was discussed.
The results from this pre-post-test design for the Teen Journey of Hope indicate the program is
reaching the goal of building knowledge and coping skills among the participants. These results
are promising given the short duration of the program (3 1-hr long sessions) and considering this
is the first time the Teen Journey of Hope has been implemented in a school setting.
23
25 participants stated they benefitted from the discussions on how to cope with stress.
33
Table 16: TJoH pre and post-test comparisons on quantitative survey
Table 17: Means of TJoH quantitative survey N Mean Pre test Mean Post-
test
Significance
Support 139 3.18 3.36 .07
Stress affects body 138 3.93 4.09 .19
Signs of stress 134 3.51 3.99 .00
Coping knowledge 137 3.23 3.86 .00
Breathing 136 3.27 3.85 .65
Manage Stress 134 3.72 3.99 .05
Type of Stress 132 2.91 3.86 .00
Social Support 139 3.68 3.81 .21
2
2.5
3
3.5
4
4.5
Signs of stress Coping Breathing Types of Stress
Pre test
Post-test
34
5.0 Key Findings and Discussion
The evaluation of the Elementary, Adolescent and Teen Journey of Hope curricula in Alabama
has a number of noteworthy findings. First, the evaluation adds to the evidence base of the
Journey of Hope’s curricula in a post-disaster setting. While the original evaluation conducted
by IICRD provided qualitative findings and the evaluation in New Zealand illustrated the
applicability of the program in an international setting, the current research enhances Save the
Children’s knowledge on the impact of the curricula in the United States. The use of multiple
data sources including validated scales, written questionnaires, individual interviews, focus
groups and district level disciplinary referrals enabled triangulation of the data. Furthermore,
using more than quantitative self-report measures strengthened the vigor of the evaluation. As
Dunning and colleagues (2004) found in a meta-analysis of self-report measures, people often
over or under-estimate their functioning and do not often accurately report the effect of emotions
such as anxiety and fear on their functioning (Dunning, Health & Suls, 2004). By gathering data
from multiple sources, the evaluators were able to identify outcomes that may not have been
captured solely from self-reported quantitative measures.
This research also suggests that the curricula are appropriate in a post-disaster setting. This is
essential because of the dearth of evidence-based psychosocial curricula after a disaster (Roberts
& Everly, 2006). Many evidence-based curricula are treatment oriented, but the Journey of
Hope curricula can be implemented with all youth who have experienced a disaster because of its
strengths-based approach to understanding common feelings and reactions after a traumatic
situation. Moreover, the Journey of Hope curricula provide a distinct contribution to addressing
the psychosocial needs of youth after a disaster. The final section of this report discusses the
implications of the evaluation for the Teen, Adolescent and Elementary Journey of Hope.
5.1 Elementary and Adolescent Journey of Hope
The evaluation of the Elementary and Adolescent Journey of Hope are valuable on multiple
levels. First, the outcomes of the research provide a more thorough evidence base for the
curricula. Additionally, the evaluation supports the applicability of the curricula in a post
disaster setting. The outcomes of the study indicate that the curricula not only provides children
35
with psycho-educational knowledge, but also helps them process emotions such as anger and
enhances peer relationships which may enrich coping skills in future adverse events.
Elementary Journey of Hope
Participating in the EJoH increased psycho-educational knowledge.
The data clearly supported that youth increased their psycho-educational knowledge. The
written questionnaires asked direct questions about what was learned in the program, and high
scores on the Likert scale (4.0-5.0) indicate that the Elementary Journey of Hope program is
meeting its objective that participants learn about various emotions that they may encounter after
a disaster. Additionally the qualitative interviews illustrated that information learned through
the program resonated within many participants, and has helped youth process knowledge about
feelings such as sadness, fear, how to deal with bullies, and how not to take their anger out on
others.
Participation in the EJoH reduces externalizing behaviors.
The reduction in externalizing behaviors was particularly noteworthy considering one symptom
of a traumatic event is acting out. When youth do not have a safe and secure outlet to process
their emotions they can become reactive to and express their emotions through anger, aggression
and bullying. Multiple data sources (written questionnaires, qualitative interviews, disciplinary
referral data, Youth Coping Inventory) supported that youth learned about how to more
effectively process their externalizing symptoms. The standardized written questionnaires
illustrated that the “learning how to handle anger” section was a topic that resonated with the
youth (4.83 out of 5.0). Additionally, the 30% reduction in disciplinary referrals indicated that
participants of the EJoH were acting out less in their classes and at school. The Youth Coping
Inventory also illustrated improvements in the experimental group of tension management and
communication skills for those who participated in the EJoH. Finally, the thematic findings from
the qualitative interviews clearly indicated that youth found ways to better handle and process
their feelings of anger and aggression.
36
Adolescent Journey of Hope
Involvement aided in building peer relationships.
Qualitative interviews clearly illustrate that students enjoyed participating in the curricula
because they were able to interact with their peers in a safe, confidential setting. Additionally,
the adolescents reported better peer relationships in the Communities that Care survey from pre
to post-test. This finding is noteworthy considering peer relationships are a protective factor
from the negative psychological impacts of a disaster. While this outcome was not significant, it
was nearing significance and should be more closely monitored in future research through
adding more peer connectedness scales as well as comparing the group who participated in the
AJoH (experimental group) to a group who has not participated in the program (control group).
By comparing the outcomes from the two groups, an “effect size” would be available to measure
the impact on those who participated in the program versus those who did not.
Participation in the curricula supported children in understanding and normalizing emotions
associated with trauma or difficult circumstances.
The significant reductions of emotional distress scores on the Strengths and Difficulties
Questionnaire among 6th
-8th
grade males and 8th grade females coupled with qualitative
interviews indicate youth learned skills to help reduce emotional distress and positively cope
with their emotions. As research on disasters such as Hurricane Katrina, the September 11th
World Trade Center attacks, the Tsunami in Indonesia and other disaster affected areas has
found, adolescents have a higher prevalence of emotional distress symptoms in the longer-term
post-disaster period if they do not receive mental health services (Agustini, Asniar, & Matsuo,
2011; McLaughlin et al., 2009; Mijanovich & Weitzman, 2010). Given this knowledge, the
evaluation provides valuable information about the AJoH’s possible impact on reducing
emotional distress. Furthermore, it lays the groundwork for a future randomized controlled study
design measuring emotional distress symptoms in youth who participate in the AJoH.
5.2 Teen JoH
This evaluation is the first to be conducted on the Teen Journey of Hope. By evaluating the
TJoH, the evaluators were able to understand the impact of this pilot curriculum. The data
presented statistically significant (p<.05) findings between pre and post-test on knowledge-based
37
questions which are key objectives of the program such as: (1) signs of stress, (2) coping (3)
breathing exercises and (4) types of stress These findings indicate the Teen Journey of Hope is
clearly reaching its audience of older teens. Moreover, programs that target older teens have
historically been a gap in the Save the Children’s psychosocial programming; therefore, this
evaluation is the first to identify the impact of an intervention for this population.
Enhanced knowledge about stressors and coping strategies
This was a crucial finding considering research has directly linked adolescents who were
exposed to trauma and left untreated at higher risk for depressive symptoms and chronic PTSD
than those who received some form of psychological support (Goenjian et al., 2005). Providing
an outlet for teens to learn about positive ways to cope with stressors may, therefore, reduce
long-term negative psychological sequela.
Increased knowledge of types of stress
This was a noteworthy outcome of the study given that many teens experience a variety of
stressors after a disaster. The Teen Journey of Hope is a program that can reduce these feelings
and help teens identify and cope with specific types of stress they may encounter.
Future research may inquire about the long-term impacts of the program and whether the teen
program helps improve overall well-being of participants after a disaster. Moreover, subsequent
research using scales that measure stress, anxiety, self-esteem and self-efficacy would enhance
the knowledge base of the TJoH and lend to a more in-depth understanding of the programmatic
impact.
5.3 Limitations
This study is a subsequent step to evaluating the efficacy of the Journey of Hope curricula.
While results support the evidence for the curricula, some limitations must be identified. First,
the waitlist-control design was only available for the students who participated in the Elementary
Journey of Hope. Future evaluations would benefit from a control group with the older students
who participated in the Adolescent and Teen Journey of Hope to compare the program effects.
Second, there were limited self-report measures for the younger youth, therefore, specific
38
outcomes such as self-efficacy, anger management, and peer relationships were not self-reported.
Teacher report measures were distributed to identify these outcomes, however, limited responses
made analysis impossible. While a strength of the evaluation was the availability of the
disciplinary referral data, other measures of externalizing behaviors would provide a more
extensive knowledge of the programmatic impact.
Another limitation was that there were only two time points measured in the evaluation. Given
the short time period of the evaluation, the researchers were unable to measure the longitudinal
effects of the curricula. A follow-up evaluation measuring three time points would help Save the
Children understand the longer term impact of the program.
5.4 Conclusions
Despite the limitations of the evaluation, the findings lend support to the evidence of the Journey
of Hope curricula. Given the abrupt nature of disasters and the breakdown of systems and
services, it is often difficult to measure the impact of a psychosocial program (Athula et al.,
2010). The current evidence-base of mental health programs for people exposed to conflict and
disasters is weak, therefore, continued research must be completed on programs such as the
Journey of Hope that address the mental health needs of youth.
Although there is a growing collection of empirical data about evidence-based programming in
post-disaster mental health, researchers have posited that continued evaluations are needed and
further evidence should be gathered through culturally appropriate research in order to
effectively respond to the affected populations. Moreover, the research needs to be appropriate
both culturally and regionally in order to adequately address the needs of those impacted by the
disaster (Athula et al., 2010). Considering the Elementary and Adolescent Journey of Hope
curricula have been evaluated in three culturally diverse areas (New Orleans, LA; Tuscaloosa,
AL; and Christchurch, NZ) and yielded promising results in each setting with minimal
adaptation, the curricula provide a unique contribution to the field. While funding has been
time-limited for each of the evaluations, they build a basis for future funding which would solely
support more controlled longitudinal research on the curricula from entities such as Substance
Abuse and Mental Health Service Administration (SAMHSA) or National Institute of Health
39
(NIH). After a natural disaster, there is a great need for evidence-based programs to address the
emotional well-being of children, adolescents, older teens and emerging adults. The Journey of
Hope curricula help fill those needs and close the gap in psychosocial programming after a
disaster.
40
Appendix 1: Teen Journey of Hope Questionnaire
Journey of Hope Questionnaire
Pre-Test
Directions: Please answer the following questions based on how you currently feel. The questions will also be asked after the session. There are no right or wrong answers but are to help Save the Children staff assess the
impact of the program. The survey is on a 5 point scale, please indicate your agreement with the statement from 1-5 (1=Not at all, 3=Somewhat, 5=Very).
Not at all Somewhat A lot
1 2 3 4 5
1 How much awareness do I have about supports in my school?
2 How much do I know about how stress affects my body?
3 How well can I identify signs of stress?
4 How well do I understand coping strategies to handle my stress?
5 What knowledge do I have about breathing and muscle relaxation to lower stress?
6 How much stress do I currently feel?
7
How often do I consider a positive future for myself?
8 How well can I identify positive ways to manage stress?
9 How well do I know the different types of stress?
10 How important do I consider social supports (friends and family) in improving my well-being?
41
Journey of Hope Questionnaire Post-Test
Directions: Please answer the following questions based on how you currently feel. There are no right or wrong answers but are to help Save the Children staff assess the impact of the program. The survey is on a 5 point scale,
please indicate your agreement with the statement from 1-5 (1=Not at all, 3=Somewhat, 5=Very).
Not at all Somewhat A lot
1 2 3 4 5
1 How much awareness do I have about supports in my school?
2 How much do I know about how stress affects my body?
3 How well can I identify signs of stress?
4 How well do I understand coping strategies to handle my stress?
5 What knowledge do I have about breathing and muscle relaxation to lower stress?
6 How much stress do I currently feel?
7
How often do I consider a positive future for myself? 8 How well can I identify positive ways to manage
stress? 9 How well do I know the different types of stress?
10
How important do I consider social supports (friends and family) in improving my well-being?
11 How comfortable did you feel participating in the activities?
12 Did this program meet your expectations?
1. What did you like best about Journey of Hope?
2. In what ways could we improve Journey of Hope?
42
Appendix 2: Strengths and Difficulties Questionnaire
For each item, please mark the box for Not True, Somewhat True or Certainly True. It would help us if you
answered all items as best you can even if you are not absolutely certain . Please give your answers on the
basis of how things have been for you over the last six months.
Not True Somewhat True Certainly True
1. I try to be nice to other people. I
care about their feelings
2. I am restless, I cannot stay still for
long
3. I get a lot of headaches, stomach-
aches or sickness
4. I usually share with others, for
example CD’s, games, food
5. I get very angry and often lose my
temper
6. I would rather be alone than with
people of my age
7. I usually do as I am told
8. I worry a lot
9. I am helpful if someone is hurt,
upset or feeling ill
10. I am constantly fidgeting or
squirming
11. I have one good friend or more
12. I fight a lot. I can make other
people do what I want
13. I am often unhappy, depressed or
tearful
43
14. Other people my age generally
like me
Not True Somewhat True Certainly True
15. I am easily distracted, I find it
difficult to concentrate
16. I am nervous in new situations. I
easily lose confidence
17. I am kind to younger children
18. I am often accused of lying or
cheating
19. Other children or young people
pick on me or bully me
20. I often offer to help others
(parents, teachers, children)
21. I think before I do things
22. I take things that are not mine
from home, school or elsewhere
23. I get along better with adults than
with people my own age
24. I have many fears, I am easily
scared
25. I finish the work I'm doing. My
attention is good
44
Appendix 3: Save the Children Scales
Elementary Journey of Hope Questionnaire
Please let us know what you thought of the program by answering the questions to the best of your ability. The first part of the form asks you to fill out the answers on a five point scale including: always, most of the time, a little, never, and I don’t know. There are no right or wrong answers we just want to see what you thought of the Journey of Hope.
Part I
Always Most of the Time
A Little Never I don’t know
1. In this group, I learned to share my feelings o o o o o 2. In this group, I liked being part of the activities
o
o
o
o
o
3. In this group, I was respected
o
o
o
o
o
4. I helped make and follow group rules
o
o
o
o
o
5. In this group, I felt I could talk to adults
o
o
o
o
o
6. In this group, I learned about what makes me feel safe
o
o
o
o
o
7. I wanted to go to the group each time
o
o
o
o
o
45
Part 2
Yes Yes, a little
Not Really
No I don’t know
1. In this group, I learned about better ways of showing my feelings
o o o o o
2. In this group, I learned how to deal with bullying
o o o o o
3. I learned it is bad to worry o o o o o
4. In this group, I learned to show my anger without hurting anybody
o o o o o
5. I did not like the games o o o o o
6. I liked the books o o o o o
7. After this group, I learned more about me o o o o o
8. I know how to make myself feel better when I’m upset
o o o o o
What did you like BEST in the group (write your answer)
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Is there anything you did NOT like? (write your answer)
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
PLEASE COMPLETE:
How old are you?________ Are you a: Boy Girl
Thank You!!
46
Appendix 4: Qualitative Interview Guide
Qualitative Interview Guide
Interview Guidelines (narrative-style interviews):
Brief explanation that we are asking them what they think of the program.
*Ask them to state their name, school and who ran the group they were a part of
1. What did you do in the group?
Probe-What did you talk about?
2. What did you like about the group/program?
Probe-What’s your favorite activity?
Probe-What’s your favorite topic?
3. What didn’t you like about it?
4. What is XX (facilitator’s name) like? (probe: Are they like your teachers?)
5. Do you think anyone else should participate in this group?
Probe-Do you think any of your friends or family should participate in this program?
Probe-(If so) Why do you think others should participate in the program?
6. What did you learn?
Probe: What, if anything, did you learn about yourself?
Probe: What did you learn about other group members?
7. Which topic was the most important to you?
Probe: Are there any feelings you still have trouble with, if so can you talk about it?
8. How comfortable did you feel sharing in the group?
Probe: Did you talk a lot or a little
9. Was there anything you didn’t talk about that you think would have helped you?
10. Do you still have your journal? Do you still look at it? Have you shown it to anyone?
11. What if anything can be improved about the Journey of Hope program?
47
Appendix 5: CTC School Bonding Survey
How much do you agree with the following statements?
NO! no yes YES!
1. I like school
2. Most mornings I look forward
to school.
3. I do extra work on my own.
4. When I have an assignment to
do, I keep working on it until its
finished.
5. I like my classes this year
48
Appendix 6: Youth Coping Inventory
Please complete the following questions to the best of your ability. Remember, there are no right or wrong answers.
Read each of the statements below that describes a behavior for coping with problems. Decide how often you do each of the described behaviors when you face difficulties or feel tense
Never Hardly ever Sometimes Often Most of
the Time
1. Apologize to People
2. Talk to a teacher or counselor at school about what bothers you
3. Read
4. Get more involved in activities at school
5. Try to improve yourself (get in shape, get better grades, etc.)
6. Try to reason with parents and talk things out; compromise
7. Try to think of the good things in your life
8. Say nice things to others
9. Get angry and yell at people
10. Work hard on schoolwork or other school projects
11. Pray
12. Try, on your own, to figure out how to deal with your problems or tensions
13. Try to make your own decisions
49
Never Hardly ever Sometimes Often Most of
the Time
14. Go to church
15. Swear
16. Organize your life and what you have to do
17. Go along with parents’ requests and rules
18. Blame others for what’s going wrong
19. Tell yourself the problem is not important
20. Try to help other people solve their problems
21. Get professional counseling (not from a school teacher or school counselor)
22. Try to keep up friendships or make new friends
23. Daydream about how you would like things to be
24. Play video games
25. Let off steam by complaining to your friends
26. Say mean things to people; be sarcastic
27. Do things with your family
28. Talk with a friend about how you feel
29. Try to see the good things in a difficult situation
30. Work on a hobby you have
31. Do a physical activity (jog, bike, etc.)
50
References
Agustini, N., Asniar, I., & Matsuo, H. (2011). The prevalence of long-term post-traumatic stress
symptoms among adolescents after the tsunami in Aceh. Journal of Psychiatric and Mental
Health Nursing, 18(6), 543-549. doi: 10.1111/j.1365-2850.2011.01702.x
Arthur, M., Hawkins, J., Pollard, J., Catalano, R. & Baglioni, A (2002). Measuring risk and protective
factors for substance use, delinquency, and other adolescent problem behaviors. Evaluation
Review 26(6), 575-601.
Athula, S., Aamir, J., Leonardo, D., Aasim, A., Darryl, M., Sandya, S., . . . Chesmal, S. (2010). Ethical
Issues in post-disaster clinical interventions and research: A developing world perspective. Key
findings from a drafting and consensus generation meeting of the working group on disaster
research and ethics (WGDRE) 2007. Asian Bioethics Review, 2(2), 124-142.
Cohen, Jaycox, Walker, Mannarino, Langley, & DuClos. (2009). Treating Traumatized Children after
Hurricane Katrina: Project Fleur-de Lis™. [Article]. Clinical Child & Family Psychology Review,
12(1), 55-64. doi: 10.1007/s10567-009-0039-2
Garrett, A., Grant, R., Madrid, P., Brito, A., Abramson, D., & Redlener, I. (2007). Children and
megadisasters: lessons learned in the new millennium. [doi: DOI: 10.1016/j.yapd.2007.03.011].
Advances in Pediatrics, 54(1), 189-214.
Goenjian, A., Walling, D., Steinberg, A., Karayan, I., Najarian, L., & Pynoos, R. (2005). A prospective
study of posttraumatic stress and depressive reactions among treated and untreated adolescents 5
years after a catastrophic disaster. American Journal of Psychiatry, 162(2302-2308).
Jaycox, L., Morse, L., Tanielian, T., Stein, B. (2006). How schools can help students recover from
traumatic experiences a tool kit for supporting long-term recovery. Arlington, VA: Rand
Kar, N. (2009). Psychological impact of disasters on children: review of assessment and interventions.
World Journal of Pediatrics, 5(1), 5-11. doi: 10.1007/s12519-009-0001-x
Kataoka, S., Rowan, B., & Hoagwood, K. (2009). Bridging the divide: In search of common ground in
mental health and education research and policy. Psychiatric Services, 60(11).
51
Lofland, J., & Lofland, L. (1995). Analyzing social settings: A guide to qualitative observation and
analysis. Belmont, CA: Wadsworth, Inc.
McLaughlin, K., Fairbank, J., Gruber, M., Jones, R., Lakoma, M., Pfefferbaum, B., . . . Kessler, R.
(2009). Serious emotional disturbance among youths exposed to Hurricane Katrina 2 years post
disaster. Journal of American Academy of Child and Adolescent Psychiatry, 48(11).
Mijanovich, T., & Weitzman, B. (2010). Disaster in context: The effects of 9/11 on youth distant from the
attacks. Community Mental Health Journal, 46(6), 601-611. doi: 10.1007/s10597-009-9240-5
NASA. (2011). Tornado track in Tuscaloosa, Alabama, from
http://earthobservatory.nasa.gov/IOTD/view.php?id=50434
Ogles, B., Lunnen, K., & Bonesteel, K. (2001). Clinical significance: History, application, and current
practice. Clinical Psychology Review, 21(3), 421-446. doi: 10.1016/s0272-7358(99)00058-6
Richards, T. J., & Richards, L. (1994). Using computers in qualitative research. In N. Denzin & Y.
Lincoln (Eds.), Handbook of qualitative research (pp. 445-462). London: Sage Publications.
Roberts, A., & Everly, G. (2006). A meta-analysis of 36 crisis intervention studies. Brief Treatment and
Crisis Intervention, 6(1). doi: 10.1093/brief-treatment/mhj006
Strauss, A. (1987). Qualitative analysis for social scientists. Cambridge: Cambridge University Press.
Walsh, F. (2007). Traumatic loss and major disasters: strengthening family and community resilience.
Fam Process, 46(2), 207-227.
Wang, Y., Nomura, Y., PatHorenczyk, R., Doppelt, O., Abramovitz, R., Brom, D., & Chemtob, C.
(2006). Association of direct exposure to terrorism, media exposure to terrorism, and other
trauma with emotional and behavioral problems in preschool children. Annals of the New York
Academy of Sciences, 1094(1), 363-368. doi: 10.1196/annals.1376.051
Williams, R., Alexander, D., Bolsover, D., & Bakke, F. (2008). Children, resilience and disasters: recent
evidence that should influence a model of psychosocial care. Current Opinion in Psychiatry,
21(4), 338-344 310.1097/YCO.1090b1013e328305b328306e328304.