Echo Interim Evaluation Findings 3 18 2010

Presented by:

Ron Schack, Ph.D.

The Charter Oak Group, LLC

Purpose of Evaluation Collect formative process information for early and

continuing improvement of the model;

Collect fidelity to model information for different implementation sites

Collect information on “how well” the model is being implemented (fidelity to model, commitment, use of different components)

Collect information on whether anyone is better off (behavior change of adults and youth)

3/16/2010 The Charter Oak Group, LLC 2

Results-Based Accountability(RBA) Model

Result : Youth achieve optimal success in school and life.

Population (Community) Indicators: % court involved, % graduating from HS, % employed

Strategies:

Increasing youth resiliency and self esteem through involvement with caring adults who convey ECHO core messages

Performance measures: % training participants rating the training as useful and relevant % students demonstrating understanding of ECHO core messages % students/athletes increasing resiliency, % students/athletes increasing self-esteem, % students/athletes reporting positive changes in teaching or coaching behavior, % students with good school attendance, % students with no post-program suspensions/expulsions


Evaluation Methodologies Observation

Participant Training Surveys

Athlete/Student Pre-Post Surveys

Outcome Measures


Matrix of Implementations/Methods


Training

Participant

Survey

Pre-Post

Athlete/Stude

nt Survey

Coaching

Observation

Protocol

Other

Observation/F

ocus Groups

Testing on

ECHO Core

Messages

Outcome

Measures

Bridgeport Summer

Basketball League (and

related leagues in

other towns)

No (was

done in

Year1)

Yes, using a

sample

approach

Yes, for

2010

Yes No No

Roberto Clemente

Leadership Academy

Yes Yes No Yes No Yes

Bassick High School Yes Yes No Yes Yes Yes

South Windsor Parks

and Recreation

No (was

done in Year

1)

Post only

(looking of

year to year

differences

Yes Yes No No

Hartford Afterschool

Programs

Yes Yes No Yes No No

Bridgeport Summer League


3.323.38

3.19

3.313.38

3.31

1

1.5

2

2.5

3

3.5

At times, I think I am no good at all

I feel that I have a number of good

qualities

I am able to do things as well as

most other people

I feel I do not have much to be proud

of

I feel useless at times

I wish I could have more respect for

myself

Pre-test

Post-test

Bassick High School


91.2%

62.0%

47.4%

73.0%

59.1%

55.5%

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0%

What is ECHO?

Example of Empathy?

Why Not Bully?

Life Goals

Actions For Goals

Strong Postive Relationships

Bassick Student Responses To Core Message Questions, % with Full Credit (N=137)

Roberto Clemente Leadership Academy


Qualitative Findings Bridgeport Summer League

Games were orderly No trash talking Coaches didn’t yell or single out individuals Youth were attentive and listened to the coaches

Coaches Appreciated ECHO support Found it difficult to take the time to have “dedicated” ECHO moments, but

embedded ECHO principles into their coaching Appreciated the more systematic and organized support provided in 2009 as

opposed to previous years Believe that the ECHO basketball league gives hope to kids, many of whom have few

positive experiences and positive role models Interns

Interns were happy to have opportunity Proud to be looked up to by younger kids Given real responsibilities


Qualitative Findings, Continued Teacher Comments On ECHO at Bassick:

ECHO puts a name to something we all do or try to do, but it is easier when you have a framework to follow.

ECHO messages need to be repeated and sustained; sometimes you may “have” a kid and then lose them, and you need to reiterate the messages.

The “moments of greatness” stickers and certificates seem to have a positive effect…they also allow the teacher to “remind” the student of prior great moments.

The monthly support is great…it helps us re-focus on ECHO and anticipate what we will do for the core messages in the upcoming month.

ECHO does not seem to work as well on the really “hard cases.” On the other hand, if you can target some of the “trouble-making”

leaders and turn them around using ECHO approaches, a lot of others will follow


Qualitative Findings, ContinuedComments From Clemente Teachers and Administrators About the

Impact of ECHO (March 1, 2010):

One teacher reported that a student in her class befriended the student who was most often teased, causing other students to take notice of the new friendship. The teacher reported a subsequent reduction in the teasing.

The librarian felt that students overall were “calmer.”

Teachers reported that the students were eager to earn the moments of greatness awards and were stepping up their acts of kindness.

Students are beginning to verbalize the core values on their own without teacher prompting


Summary of Key Findings Participant training surveys scores have improved since

early implementation; participants find on-going technical assistance helpful and important.

There are statistically significant increases on some important self-esteem questions.

Flexible adaptability of ECHO makes it more likely to implemented; intensity of implementation matters to results but is not the only factor.

Continuing commitment to ECHO by executive leadership at implementation sites, even when alternatives exist, shows that ECHO has provided real added value



1 1.5 2 2.5 3 3.5 4

Summer League Pre

Bassick Post

Clemente Mid-Year

Summer League Pre

Bassick Post

Clemente Mid-Year

Summer League Pre

Bassick Post

Clemente Mid-Year

Summer League Pre

Bassick Post

Clemente Mid-Year

Summer League Pre

Bassick Post

Clemente Mid-Year

Summer League Pre

Bassick Post

Clemente Mid-Year

Summer League Pre

Bassick Post

Clemente Mid-Year

At

tim

es, I

th

ink

I am

no

go

od

at

all

I fee

l th

at I

hav

e a

nu

mb

er

of

goo

d

qu

alit

ies

I am

ab

le

to d

o

thin

gs a

s w

ell a

s m

ost

o

ther

p

eop

le

I fee

l I d

o

no

t h

ave

mu

ch t

o

be

pro

ud

o

f

I fee

l u

sele

ss

at t

imes

I wis

h I

cou

ld

hav

e m

ore

re

spec

t fo

r m

ysel

f

I tak

e a

po

siti

ve

atti

tud

e to

war

d

mys

elf

Significant Positive Differences In Self-Esteem, Pre (Bridgeport Summer League (N=314)), Post (Bassick (N=134) and Roberto Clemente (N=121))--Significant differences in Red

Next Research Steps School-based outcomes; use of comparison groups

Observations of coaching behavior (UCONN)

Year-t0-year resiliency scores

Self-esteem, resiliency, and outcomes by age and type of youth

Changes in fitness/nutrition behavior following implementation of health/wellness component


Echo Interim Evaluation Findings 3 18 2010

Documents

Transcript of Echo Interim Evaluation Findings 3 18 2010