Download - Impact of Alchemy, Inc. on Participants’ Social …alchemyinc.net/wp-content/uploads/2016/01/SEL_Evaluation...Impact of Alchemy, Inc. on Participants’ Social Emotional Learning

Impact of Alchemy, Inc. on Participants’ Social Emotional Learning

2014-2015 School Year

The Research and Evaluation Bureau, Kent State University

Table of ContentsIntroduction…………………….…… 3-4

Evaluation Focus and Design….….... 5-8

Participants………………….………. 9-11

Findings……………………………… 12-55

Open-Ended Questions……….……... 56-63

Summary…………………………….. 64-70

Conclusions…………………….……..Recommendations……………………

71-7273

2

3

About Alchemy, Inc.

• Alchemy, Inc. is a non-profit organization with a mission to assist urban adolescent males to develop a sense of purpose in life and successfully function as members of a family, school, and community.

• Alchemy’s name is derived from the ancient art of transformation in which medieval alchemists attempted to transform a lower base metal, or prima materia, into gold, called the Philosopher’s Stone. Alchemy, Inc. works to transform prima materia (urban youth) into the Philosopher’s Stone (the gold inherent within our youth).

• The goals of Alchemy, Inc. include the following: o Increase Social Emotional Learning (SEL) o Increase school attendance o Increase GPA averages o Increase high school graduation rates o Increase college acceptance rates o Develop character traits necessary to succeed in life

4

The Sacred Circle of Alchemy, Inc.

• The program uses myth as a sociological and pedagogical tool with the goal of teaching participants how to live in the world together. o The myths also provide examples of how each participant can realize

the personal gifts that he brings to the world.

• Myths are told to the beat of an African drum with the aim of creating a safe environment and sense of community for participants.

• The Sacred Circle of Alchemy, Inc. is derived from the ritual of Aesculapius and other traditional cultures. o Youth are greeted by the adults present and then relate a brief history

of their ailment through the discussion of myth. o Hand sanitizer of some sort is passed around to symbolize washing. o The drumming is reminiscent of the cultures’ sacred song-chanting.o Youth come to the circles to be healed through their analysis and

discussion of myths.

5

Evaluation Focus

• The focus of this evaluation is to assess the Social Emotional Learning (SEL) growth from participation in Alchemy, Inc.

• SEL is defined as the process by which students learn to: o Recognize and manage emotionso Care about otherso Make good decisionso Behave ethically and responsiblyo Develop positive relationshipso Avoid negative behavior

• SEL enhances the student’s ability to integrate thinking, feeling, and behaving in order to achieve important life tasks.

• The major benefits of SEL are to:o Enhance growth and learning o Prepare students for life

6

Evaluation Design

• A survey was administered to 6th through 12th grade participants of the Alchemy, Inc. program in the following schools:

o Crouse Middle School o Innes Middle School o Litchfield Middle School o Akron Early College High School o Buchtel Community Learning Center High School o Copley High School o Coventry High School o Firestone High School o Garfield High School o Kenmore High School o St. Vincent-St Mary High School

Evaluation Design Continued…

• The survey presented participants with major themes faced by the heroes in the myths. The themes represent challenges in the hero’s quest. For example, one theme is “being connected to something larger than oneself.”

• Within each theme, there were five to seven pairs of Likert-type items representing social-emotional development concepts.

• The first item in each pair asked the participant to indicate the extent to which he felt he had learned about that concept. The second item in each pair asked the participant to indicate the extent to which he felt his behavior or attitude had changed in that area as a result of participation in Alchemy, Inc.

• Participants responded to each item using the following scale: 1=Not at all 2=Only a little 3=Some 4=A lot

• A total of 60 surveys were collected.7

8

Evaluation Design Continued…

• Open-ended questions were included in the survey to assess the following:o Which myths made the biggest impact on participants in terms of each

major theme presentedo Who are the most influential males in the lives of participantso Which myth had the biggest impact overall on participantso Changes that participants have made in their lives as a result of

participation in Alchemy, Inc.o What participants like most about Alchemy, Inc.

9

ParticipantsNumber of Students in Each Grade Level at Each Participating School

6th 9th 10th 11th Total

Innes Middle School 10 0 0 0 10

Crouse Middle School 13 0 0 0 13

Litchfield Middle School 14 0 0 0 14

St. Vincent-St. Mary High School 0 3 0 0 3

Buchtel CLC High School 0 5 1 0 6

Firestone High School 0 0 3 1 4

Kenmore High School 0 2 1 0 3

Akron Early College High School

0 1 0 0 1

Coventry High School 0 1 0 0 1

Garfield High School 0 1 2 1 4

Copley High School 0 1 0 0 1

*No participants from 7th, 8th, and 12th grade

Participants continued…

Grade N Percent

6th 37 62%

9th 14 23%

10th 7 12%

11th 2 3%

Total 60 100%

Summary of Number of Students in Each Grade Level

10

Participants continued…

Number of Years N Percent

1 37 62%

2 3 5%

3 0 0%

4 or More 20 33%

Total 60 100%

10

Including this year, how long have you been participating in Alchemy?

12

Findings

• The survey was divided into sections corresponding to the major themes faced by the heroes in the myths.

• Participants responded according to the extent they had “learned about” each item and then according to the extent to which they had “changed their behavior/attitude” about each item.

• The following slides present descriptive statistics (percentages and averages) for each item within each section.

13

Section 1Being Connected to Something

Larger than Oneself

14

Recognizing the rights of others

n=23; mean=3.65

Scale: 1=Not at all, 2=Only a little, 3=Some, 4=A lot

0

20

40

60

80

100

69.6

26.1

4.3

0.0

Learned

Per

cent

0

20

40

60

80

100

68.2

31.8

0.0 0.0

Behavior or Attitude Changed

Per

cent

n=22; mean=3.68

15

Respecting the points of view of others

n=22; mean=3.59


0

20

40

60

80

100

59.1

40.9

0.0 0.0

Learned

Per

cent

0

20

40

60

80

100

63.6

27.3

9.1

0.0


Per

cent

n=22; mean=3.55

16

Accepting and respecting individual differences

n=22; mean=3.68


0

20

40

60

80

100

72.7

22.7

4.5

0.0

Learned

Per

cent

0

20

40

60

80

100

68.2

22.7

9.1

0.0


Per

cent

n=22; mean=3.59

17

Accepting and respecting ethnic and cultural differences

n=23; mean=3.48


0

20

40

60

80

100

52.2

43.5

4.3

0.0

Learned

Per

cent

0

20

40

60

80

100

63.6

27.3

4.5 4.5


Per

cent

n=22; mean=3.50

18

Which myth made the biggest impact on you in terms of learning about becoming connected?

• The Devil’s Sooty Brother (4)• Parsifal (3)• The Firebird (2)• The Killing Virtue (1)• The Water of Life (1)• Go I Know Not Whither, Bring Back I Know Not What (1)• Theseus (1)• The King of the Golden Mountain (1)• The White Snake (1)• The Golden Bird/Feather (1)• Love Caused It All (1)

Number in parentheses indicates number of responses for each item.

19

Section 2Acquiring Basic Skills

20

Learning how to set goals

n=23; mean=3.57


0

20

40

60

80

100

69.6

21.7

4.3 4.3

Learned

Per

cent

0

20

40

60

80

100

60.9

34.8

4.3

0.0


Per

cent

n=23; mean=3.57

21

Identifying skills I need

n=23; mean=3.48


0

20

40

60

80

100

47.9 52.2

0.0 0.0

Learned

Per

cent

0

20

40

60

80

100

65.2

26.1

8.7

0.0


Per

cent

n=23; mean=3.57

22

Finding new ways to solve problems

n=21; mean=3.62


0

20

40

60

80

100

66.7

28.6

4.8

0.0

Learned

Per

cent

0

20

40

60

80

100

47.8 47.8

4.3

0.0


Per

cent

n=23; mean=3.43

23

Developing coping skills to deal with problems

n=23; mean=3.04


0

20

40

60

80

100

30.4

43.5

26.1

0.0

Learned

Per

cent

0

20

40

60

80

100

40.945.5

9.14.5


Per

cent

n=22; mean=3.23

24

Not giving up in learning new skills

n=22; mean=3.82


0

20

40

60

80

100

81.8

18.2

0.0 0.0

Learned

Per

cent

0

20

40

60

80

100

86.4

13.6

0.0 0.0


Per

cent

n=22; mean=3.86

25

Searching for helpful resources

n=23; mean=3.61


0

20

40

60

80

100

60.9

39.1

0.0 0.0

Learned

Per

cent

0

20

40

60

80

100

82.6

13.0

4.3

0.0


Per

cent

n=23; mean=3.78

26

Which myth made the biggest impact on you in terms of learning about acquiring basic skills?

• Killing Virtue (2)• King of the Golden Mountain (2)• The Devil’s Sooty Brother (2)• Hunter and His Son (1)• Parsifal (1)• Woman in the Woods (1)


27

Section 3Forming Relationships with Peers,

Family, and the Community

28

Learning how to make and keep friends

n=23; mean=3.09


0

20

40

60

80

100

30.4

52.2

13.04.3

Learned

Per

cent

0

20

40

60

80

100

56.5

34.8

8.7

0.0


Per

cent

n=23; mean=3.48

29

Recognizing the feelings of others

n=22; mean=3.32


0

20

40

60

80

100

45.5 40.9

13.6

0.0

Learned

Per

cent

0

20

40

60

80

100

60.9

26.1

13.0

0.0


Per

cent

n=23; mean=3.48

30

Using effective communication skills

n=23; mean=3.26


0

20

40

60

80

100

39.1

47.8

13.0

0.0

Learned

Per

cent

0

20

40

60

80

100

50.0

36.4

13.6

0.0


Per

cent

n=22; mean=3.36

31

Listening to others

n=22; mean=3.50


0

20

40

60

80

100

63.6

22.713.6

0.0

Learned

Per

cent

0

20

40

60

80

100

69.6

26.1

4.3

0.0


Per

cent

n=23; mean=3.65

32

Being considerate of others

n=22; mean=3.50


0

20

40

60

80

100

63.6

22.713.6

0.0

Learned

Per

cent

0

20

40

60

80

100

65.2

26.1

8.7

0.0


Per

cent

n=23; mean=3.57

33

Which myth made the biggest impact on you in terms of learning about forming relationships?

• The Water of Life (4)• The Killing Virtue (2)• The Fire Bird (2)• Parsifal (1)• Go I Know Not Whither, Bring Back I Know Not What (1)• The Devil’s Sooty Brother (1)• The Fisher and the King (1)• The Hunter and Boa Constrictor (1)• Podare (1)


34

Section 4Charting a Safe Course

35

Making safe and healthy choices

n=23; mean=3.65


0

20

40

60

80

100

69.6

26.1

4.3

0.0

Learned

Per

cent

0

20

40

60

80

100

69.6

26.1

4.3

0.0


Per

cent

n=23; mean=3.65

36

Setting my own boundaries

n=22; mean=3.5


0

20

40

60

80

100

63.6

22.713.6

0.0

Learned

Per

cent

0

20

40

60

80

100

52.2

39.1

8.7

0.0


Per

cent

n=23; mean=3.43

37

Coming up with ways to achieve goals

n=23; mean=3.57


0

20

40

60

80

100

65.2

26.1

8.7

0.0

Learned

Per

cent

0

20

40

60

80

100

73.9

21.7

4.3

0.0


Per

cent

n=23; mean=3.70

38

Knowing what my strengths are

n=23; mean=3.39


0

20

40

60

80

100

47.8 43.5

8.7

0.0

Learned

Per

cent

0

20

40

60

80

100

60.9

26.1

13.0

0.0


Per

cent

n=21; mean=3.48

39

Knowing what my values and beliefs are

n=22; mean=3.50


0

20

40

60

80

100

54.5

40.9

4.5

0.0

Learned

Per

cent

0

20

40

60

80

100

65.2

26.1

8.7

0.0


Per

cent

n=23; mean=3.57

40

Dealing with peer pressure

n=23; mean=3.00


0

20

40

60

80

100

47.8

17.4 21.713.0

Learned

Per

cent

0

20

40

60

80

100

47.8

34.8

8.7 8.7


Per

cent

n=23; mean=3.22

41

Which myth made the biggest impact on you in terms of learning about charting a safe course?

• Parsifal (3)• The Water of Life (2)• The Golden Bird/Feather (2)• The Firebird (1)• Fedot’s Wife (1)• The Hunter and Boa Constrictor (1)• Theseus (1)


42

Section 5Living in Multiple Worlds

43

Identifying my feelings

n=23; mean=3.22


0

20

40

60

80

100

43.534.8

21.7

0.0

Learned

Per

cent

0

20

40

60

80

100

34.8

52.2

13.0

0.0


Per

cent

n=23; mean=3.22

44

Expressing my feelings appropriately

n=23; mean=2.91


0

20

40

60

80

100

34.8 30.4 26.1

8.7

Learned

Per

cent

0

20

40

60

80

100

30.4 30.434.8

4.3


Per

cent

n=23; mean=2.87

45

Dealing with stress and conflict

n=23; mean=3.00


0

20

40

60

80

100

30.4

43.5

21.7

4.3

Learned

Per

cent

0

20

40

60

80

100

39.143.5

13.0

4.3


Per

cent

n=23; mean=3.17

46

Being cooperative in groups

n=22; mean=3.59


0

20

40

60

80

100

77.3

4.5

18.2

0.0

Learned

Per

cent

0

20

40

60

80

100

82.6

8.74.3 4.3


Per

cent

n=23; mean=3.70

47

Adjusting to different situations

n=23; mean=3.13


0

20

40

60

80

100

34.8

52.2

4.3 8.7

Learned

Per

cent

0

20

40

60

80

100

56.5

30.4

13.0

0.0


Per

cent

n=23; mean=3.43

48

Which myth made the biggest impact on you in terms of learning about living in multiple worlds?

• The Killing Virtue (4)• Parsifal (3)• The Devil’s Sooty Brother (3)• The Fisher and the King (1)• The Water of Life (1)• Theseus (1)


49

Section 6Taking Responsibility for Oneself

50

Knowing the difference between appropriate and inappropriate behavior

n=23; mean=3.65


0

20

40

60

80

100

78.3

13.04.3 4.3

Learned

Per

cent

0

20

40

60

80

100

69.6

26.1

4.3

0.0


Per

cent

n=23; mean=3.65

51

Understanding that self-control is important

n=23; mean=3.65


0

20

40

60

80

100

69.6

26.1

4.3

0.0

Learned

Per

cent

0

20

40

60

80

100

72.7

18.2

9.1

0.0


Per

cent

n=22; mean=3.64

52

Understanding the consequences of my decisions

n=23; mean=3.57


0

20

40

60

80

100

69.6

17.4 13.0

0.0

Learned

Per

cent

0

20

40

60

80

100

78.3

17.4

4.3

0.0


Per

cent

n=23; mean=3.74

53

Knowing how to seek help when needed

n=23; mean=3.78


0

20

40

60

80

100

82.6

13.04.3

0.0

Learned

Per

cent

0

20

40

60

80

100

56.5

30.4

4.38.7


Per

cent

n=23; mean=3.35

54

Recognizing how my choices impact others

n=23; mean=3.52


0

20

40

60

80

100

56.5

39.1

4.3

0.0

Learned

Per

cent

0

20

40

60

80

100

78.3

21.7

0.0 0.0


Per

cent

n=23; mean=3.78

55

Which myth made the biggest impact on you in terms of learning about taking responsibility?

• The Water of Life (3)• The Hunter and His Son (3)• The Young Giant (2)• Jack and the Bean Stalk (1)• The Devil’s Sooty Brother (1)• Podare (1)• Firebird (1)


56

Responses toOpen-Ended Questions

57

Please list the most influential males in your life.

Note: Participants were permitted to list more than one.

Most Influential Male Number of Responses

Alchemy instructors 27

Father 18

Grandfather 8

Friends/Cousins 8

Older Brother 7

Uncle 6

Step-Father 3

Pastor 1

58

Which myth had the biggest impact on you overall? Why?

• The Water of Life (23)o “Water of life because it gives you true life meaning.”

o “The water of life because it tells me to follow my own choices.”

• The Firebird (12)• “The firebird reminded me that if you work hard you will eventually reward yourself.”

• “The firebird because it taught me that I can get through everything.”

o The Devil’s Sooty Brother (5)o “The Devil’s Sooty Brother because it really makes me think about where I was in life and

what I need to do to correct my wrongs.”

• The Killing Virtue (3)o “The killing virtue because when his friend died he killed his self for his own friend.”

• Golden Bird/Feather (3)o “The golden bird because it showed what true bravery was.”.

• Parsifal (2)• “Because he did what he was destined to do and he followed his dreams.”


(Continued on next slide)

59

Which myth had the biggest impact on you overall? Why?

• Hunter and Boa Constrictor (1)• “The hunter and the boa constrictor because he shows you how greedy people think and what

people don’t look at in the long road.”

• White Snake (1)• “The White Snake because I learned how to pay attention more”

• Theseus (1)• “Theseus, because finding a way to do something that is important to you can help you with

other things.”

• Fidot’s Wife (1)• “Fidot’s wife because it showed me that anything can happen.”

• The Hunter and His Son (1)• “The Hunter and His Son because I did some stuff that my dad really wanted me not to do and

it really hurt him”

• The Fisher and the King (1)*

• King of the Golden Mountain (1)** Indicates that the participant(s) did to specify why the myth had the biggest impact


60

List changes you have made in your life due to participating in Alchemy.

• Interpersonal Skillso Greater respect for others (11)

o “I learned how to respect more.”o “Respecting everybody no matter what their abilities.”

o Listens to others more (8)o “I listen more.”o “Listen to other’s opinions.”

o Improved communication skills (6)o “Speak out more.”o “I have learned to talk about my behavior and not to lie.”

o Decreased conflict with others (4)o “I forgave my dad…”o “Learned how to get along with others.”

Note: Open-ended responses were provided to the above question: “List changes you have made in your life due to participating in Alchemy”. The evaluators categorized student responses into specific categories.

Number in parentheses indicates number of times this concept was stated by student participants.


• Academicso Greater focus on academics/ has improved grades (26)

o “I got honor and merit now.”o “I got my grades up...”

o Focuses on tasks (7)o “I have been working more on achieving a more peaceful and

tranquil state of mind…”

• Othero Planning for the future (2)

o “Making me think more about life challenges.”

60Note: Open-ended responses were provided to the above question: “List changes you have made in your life

due to participating in Alchemy”. The evaluators categorized student responses into specific categories.Number in parentheses indicates number of times this concept was stated by student participants.


• Social Emotional Learningo Improved behavior (21)

o “I have stopped smoking and drinking.” o “I no longer get rude to teachers.”

o Improved attitude (14)o “I changed my attitude towards things.”o “I also learned you always need a little help in life.”

o Thinks before acting (5)o “Making smarter choices.”o “I have made smarter choices in my life. Like, I think before I do

stuff.” o More responsible (3)

o “I do my homework more.”o “I have increased my work ethic.” 61

Note: Open-ended responses were provided to the above question: “List changes you have made in your life due to participating in Alchemy”. The evaluators categorized student responses into specific categories.

Number in parentheses indicates number of times this concept was stated by student participants.

63

What do you like most about Alchemy?• The myths/stories (15)

• “Able to talk and be open and also listened to” (9)

• Brotherhood/family (9)

• Teachings (8)

• Meeting everyone monthly (7)

• Fun activities (3)

• Drumming (1)

• Environment (1)

• Getting Paid (1)


64

Summary

• 100% of the students stated that they learned the following behaviors “a lot” or to “some” extent while in Alchemy:• Respecting the points of view of others

• Identifying skills I need

• Not giving up in learning new skills

• Searching for helpful resources

• Less than 75% of the students reported that they learned the following behaviors “a lot” or to “some” extent while in Alchemy:• Developing coping skills to deal with problems

• Dealing with peer pressure

• Expressing my feelings appropriately

• Dealing with stress and conflict

65

Summary

• 100 % of the students stated that their behavior or attitude changed “a lot” or to “some” extent in the following categories while in Alchemy:• Recognizing the rights of others

• Not giving up in learning new skills

• Recognizing how my choices impact others

• Less than 75% of the students stated that their behavior or attitude changed “a lot” or to “some” extent in the following categories while in Alchemy:• Expressing my feelings appropriately

66

Summary

o Survey items pertaining to youth development concepts were posed in pairs: To what extent students learned that concept and to what extent students’ behavior regarding that concept changed.

o Graphs illustrating the findings for each pair were presented above.

o A qualitative examination of the pairs revealed that at times, students reported that they learned the concept more than they reported that their behavior changed, and at other times, just the opposite.

o The following slides present a summary of the most noteworthy discrepancies between the extent to which students learned a concept and the extent to which students’ behavior changed regarding that concept.

Summary: The extent learned was greater than the extent to which behavior changed

• Respecting the points of view of others• Recognizing the feelings of others• Being considerate of others• Identifying skills I need

68

Summary: The extent to which the behavior changed was greater than the extent learned

• Dealing with peer pressure • Developing coping skills to deal with problems • Being cooperative in groups • Listening to others

69

69

Summary: Perceptions of Learning

90-100% 80-89% 65-79%0

5

10

15

20

25

30

16

11

4

Summary of Number of Items Related to Learning that Participants Rated “A Lot” or “Some”

Percentage of Students Reporting They Learned “A Lot” or “Some”

Num

ber

of It

ems

70

Summary: Perceptions of Behavior/Attitude Change

90-100% 80-89% 65-79%0

5

10

15

20

25

30

0

11

4

Summary of Number of Items Related to Behavior/Attitude Change that Participants Rated “A Lot” or “Some”

Percentage of Students Reporting They Learned “A Lot” or “Some”

Num

ber of

Item

s

71

Conclusions

• Students stated that the program has helped them do better academically, demonstrate improved behavior, and have greater respect for others.

• Alchemy instructors were cited equal with fathers as being the most influential males in their lives.

72

Conclusions

• Participants in Alchemy, Inc. generally indicated positive social emotional development in all of the items surveyed.

• In terms of social emotional learning, Alchemy, Inc. appears to be a very effective program.

73

Recommendations

• Based on survey findings, areas for further development in the future include the following:

o Implementing behavior change in regards to developing coping skills to deal with problems

o Maintaining friendships and relationshipso Identifying and expressing feelings and recognizing feelings of

otherso Dealing with stress and conflict, including peer pressure

Prepared by:

Danielle Rupprecht, Graduate Assistant

Deepa Shanadi, M.P.H, M.S., Evaluator

Deborah Shama-Davis, Ph.D., Director of College Services

Research and Evaluation Bureau

Kent State University

December 2015

Contact info:

[email protected]

[email protected]

mailto:[email protected]

mailto:[email protected]