University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice...

University of WisconsinPOPULATION HEALTH INSTITUTETranslating Research into Policy and Practice

Evaluating Wisconsin’s Safe and Supportive Schools Program

D. Paul Moberg, PhD

Daphne Kuo, PhD

University of Wisconsin-Madison

Population Health InstituteDecember 7, 2012


Overview of S3 Evaluation

Evaluation Goals•Identify areas of relative safety and support (student perceptions) in each participating school•Document improvement over time in seven areas of school safety and support, and in academic and behavioral outcomes •Identify, code and describe intervention strategies (including curriculum, programs, and policies)•Evaluate effectiveness of the program--explore the relationships between intervention strategies, student achievement, behaviors (suspension and expulsion rates), and school climate (including the seven dimensions of school safety scores).


Evaluation Approach

Useful--Support program decision-making, improvement and accountability (advice?)

Evaluation Design•Change over time within S3 schools and in aggregate•Comparison high schools (available indicators-- suspension, expulsion, graduation, achievement test scores; oYRBS where available)•Complex statistical modeling—students nested in schools, controls for population characteristics; mediators and moderators; propensity scores


Research Design

2010 2011 2012 2013 2014

S3 schools (N=56+)

Osch0 T1 O1 T2 O2 T3 O3 T4 O4

Non-S3 schools

Osch0 Osch

1 Osch2 Osch

3 Osch4


Process Evaluation

Process Measures• Annual Reports from S3 schools

• Changes in school policy • School programs implemented

• PBIS Implementation?

Would like Input:•What data are available on implementation with minimal burden to school staff ?•Should annual report be supplemented with a survey?


Outcome Data

Measures:•DoE Required S3 Index—”School Safety Score”•oYRBS—S3 Module (annual over 4 years)•Student Achievement (test scores, graduation)* •Behavioral Outcomes: (Suspensions, expulsions, behavioral incidents)*

*WINSS trends over 6 years (2 before S3 baseline) measured at the school level


How do S3 Students compare to random sample of Wisconsin Students?

Comparisons between S3 Schools and Wisconsin High Schools, Common Questions in OYRBS and YRBS 2011

Questions S3 WI

(the past 30 days) How many days did you not go to school because you felt you would be unsafe at school or on your way to or from school? 8% 3%

(the past 30 days) How many days did you carry a weapon such as a gun, knife, or club on school property? 6% 3%

(the past 12 months) How many times has someone threatened or injured you with a weapon such as a gun, knife, or club on school property? 8% 5%

(the past 12 months) How many times were you in a physical fight on school property? 17% 9%

(the past 12 months) Have you ever been bullied on school property? 20% 27%

(the past 12 months) Have you ever been electronically bullied? 15% 17%

(the past 30 days) How many days did you have at least one drink of alcohol? 31% 39%

(the past 30 days) How many days did you have 5 or more drinks of alcohol in a row, that is, within a couple of hours? 17% 24%

(the past 30 days) How many times did you use marijuana? 23% 22%

(the past 12 months) Has anyone offered, sold, or given you an illegal drug on school property? 23% 21%* Frequencies for Wisconsin YRBS are from CDC online analysis of about 1400 9th and 11th grade students, does include some S3 schools.

* Frequencies for S3 High schools are from Wisconsin online YRBS survey designed for S3 project.


Suspensions in S3 Schools

Expulsion and Suspension Rates for Participating 66 Wisconsin High Schools

Mean S.D. Median Minimum Maximum

Suspension (persons)

3 year average (08-11) .251 .203 .162 .018 .729

2010-2011 .221 .172 .164 .008 .678

Suspension (incidents)

3 year average (08-11) .669 .759 .350 .020 3.370

2010-2011 .516 .511 .315 .010 2.010

Expulsion

3 year average (08-11) .007 .007 .004 .000 .029

2010-2011 .006 .008 .004 .000 .030

* The 3 years average rates were calculated by the mean of explusion/suspension rates of year 2008-09, 2009-10, and 2010-11.


Suspension Rates in S3 Schools


oYRBS Data

Factor Analysis 7 Indicesviolence bullying and harassmentalcohol and drug userule and discipline enforcement general climate and support general safety

Caution—These are student self reported behaviors and perceptions


oYRBS S3 Factors (1)Violence10 During the past 30 days, on how many days did you not go to school because you felt you would be unsafe at school or on

your way to or from school?11 During the past 30 days, on how many days did you carry a weapon such as a gun, knife, or club on school property?12 During the past 30 days, on how many days did you carry a gun on school property?13 During the past 12 months, how many times has someone threatened or injured you with a weapon such as a gun, knife, or

club on school property?14 During the past 12 months, how many times were you in a physical fight on school property?32 During the past 30 days, how many times have you taken an over-the-counter drug to get high?Alcohol & drugs31 During the past 30 days, how many times did you use marijuana?33 During the past 12 months how many times have you attended school under the influence of alcohol or other illegal drugs,

such as marijuana or cocaine28 Drank alcohol in past 30 days. This includes drinking beer, wine, wine coolers, and liquor such as rum, gin, vodka, or

whiskey. For these questions, drinking alcohol does not include drinking a few sips of wine for religious purpose29 During the past 30 days, on how many days did you have 5 or more drinks of alcohol in a row, that is, within a couple of

hours?


oYRBS S3 Factors (2)Bullying and harrassment

18 (Have you been bullied....) Bullying is when one or more students tease, threaten, spread rumors about, hit, shove, or hurt another student over and over again. It is not bullying when two students of about the same strength or power argue.

19 During the past 12 months, have you ever been electronically bullied?(Include through e-mail, chat rooms, instant messaging, websites, or texting.)

20 During the past 12 months, have you bullied someone else on school property?

21 Harassment is when one or more students bother another student with the goal to annoy, disturb, or upset that student. During the past 12 months, have you ever been harassed on school property?

22 During the past 12 months, have you been harassed on school property because of your race or ethnic background?

23 During the past 12 months, have you been harassed on school property because of your weight, size, or physical appearance?

25 During the past 12 months, have you been harassed on school property because someone thought you were gay, lesbian, or bisexual?

24 During the past 12 months, have you been harassed on school property because of your gender?

General safety

17 Do you agree or disagree that violence is a problem at your school?

26 Do you agree or disagree that harassment and bullying by other students is a problem at your school?

27 Is there gang activity in your school?

45 I feel safe at my school

16 When you are at school, how often do you feel safe from physical harm?


oYRBS S3 Factors (3)School climate

35 Do you agree or disagree that your teachers really care about you and give you a lot of encouragement?

36 Is there at least one teacher or other adult in this school that you can talk to if you have a problem?

37 Do you agree or disagree that you feel like you belong at this school?

38 In my school, students have lots of chances to help decide things like class activities and rules.

39 There are lots of chances for students in my school to talk with a teacher one-on-one.

40 Teachers ask me to work on special classroom projects.

41 There are lots of chances for students in my school to get involved in sports, clubs, and other activities outside of class.

42 There are lots of chances to be part of class discussions or activities.

43 My teacher(s) notices when I am doing a good job and lets me know about it.

44 The school lets my parents know when I have done something well.

46 My teachers praise me when I work hard in school.

53 Do you agree or disagree that this school has a friendly and welcoming atmosphere?


oYRBS S3 Factors (4)

Rule Enforcement

54 Do you agree or disagree that you understand the rules for student behavior and conduct at this school?

55 Do you agree or disagree that this school has clear consequences for breaking the rules?

56 Do you agree or disagree that staff at this school enforce the rules for student behavior and conduct?

Importance of School Work

47 How often do you feel that the school work you are assigned is meaningful and important?

48 How interesting are most of your courses to you?

49 How important do you think the things you are learning in school are going to be for your later life?

50 Over the past school year how often did you enjoy being at school?

51 Over the past school year how often did you hate being at school?

52 Over the past school year how often did you try to do your best work in school?


Perceptions of Safe and Supportive School Environment 9th and 11th graders in 2012 Wisconsin S3 schools--oYRBS

Variable N Mean sdDPI items 21159 0.47 0.11School Climate 20698 0.52 0.13Violence (7 items) 21441 0.20 0.10Violence (6 items) 21476 0.20 0.09Bully/Harassment 21366 0.49 0.12Alcohol/Drug Use 21461 0.23 0.15Importance of School Work 21054 0.55 0.15General Safety 21311 0.60 0.15Discipline/Rules 20959 0.49 0.16Total 19718 0.44 0.09


Little difference by Gender and Grade


oYRBS Indices by Race/EthnicityHigh school students' perceptions on safe and supportive school environment, 9th and 11th graders in 2012 Wisconsin S3 schools

NH White NH Blackp1

Hispanicp1

NH Native Am.p1

Other NH groupsp1Variable N Mean sd N Mean sd N Mean sd N Mean sd N Mean sd

DPI items 9218 0.46 0.10 4622 0.48 0.10***

4243 0.49 0.11***

320 0.50 0.11***

2402 0.48 0.11***

School Climate 9036 0.51 0.13 4480 0.53 0.14***

4180 0.52 0.14***

320 0.55 0.14***

2340 0.52 0.13***

Violence (7 items) 9300 0.19 0.08 4675 0.21 0.09***

4337 0.21 0.12***

329 0.23 0.13***

2431 0.20 0.12***

Violence (6 items) 9314 0.19 0.07 4682 0.21 0.09***

4345 0.21 0.12***

330 0.23 0.13***

2436 0.20 0.11***

Bully/Harassment 9259 0.49 0.12 4668 0.47 0.10***

4315 0.49 0.12

326 0.51 0.14***

2434 0.50 0.12***

Alcohol/Drug Use 9307 0.23 0.15 4669 0.23 0.14

4342 0.25 0.17***

328 0.29 0.19***

2438 0.23 0.15

Importance of School Work 9186 0.55 0.15 4528 0.54 0.14***

4266 0.55 0.15

322 0.58 0.15***

2386 0.54 0.15***

General Safety 9253 0.57 0.14 4647 0.62 0.15***

4303 0.60 0.15***

329 0.60 0.15***

2414 0.61 0.14***

Discipline/Rules 9130 0.47 0.15 4530 0.50 0.17***

4237 0.50 0.17***

318 0.52 0.17***

2379 0.49 0.16***

Total 8702 0.43 0.08 4221 0.44 0.08***

3957 0.44 0.09***

297 0.46 0.10***

2230 0.44 0.09***

Note: The p-values indicates the degree of statistical signficance in differences between the respective racial/ethnic minority students and the non-Hispanic white students. *** p<.001, ** p<.01, * p< .05.

Note: "Other NH groups" included Asian American, Pacific Islanders, Mixed race, and other minority students. About 2000 students who did not identify race and ethnicity were excluded from this tables.


Academic Performance (n~21,000)


Academic PerformanceRelative to students of A’s


oYRBS Indices by Sexual OrientationHigh school students' perceptions on safe and supportive school environment, 9th and 11th graders in 2012 Wisconsin S3 schools

Perceptions on safe and supportive school environment by sexual orientation

Heterosexual LGB/Not Sure No answers

Variable N Mean sd N Mean sd N Mean sd

DPI items 17757 0.47 0.10 2205 0.53 0.14 1197 0.48 0.10

School Climate 17361 0.51 0.13 2155 0.56 0.17 1182 0.52 0.14

Violence (7 items) 17977 0.19 0.08 2235 0.26 0.24 1229 0.20 0.09

Violence (6 items) 17999 0.19 0.08 2243 0.25 0.24 1234 0.20 0.09

Bully/Harassment 17917 0.48 0.11 2226 0.57 0.18 1223 0.50 0.13

Alcohol/Drug Use 17959 0.23 0.14 2254 0.29 0.27 1248 0.22 0.15

Importance of School Work 17626 0.54 0.15 2194 0.59 0.16 1234 0.56 0.15

General Safety 17848 0.59 0.15 2234 0.64 0.17 1229 0.60 0.14

Discipline/Rules 17538 0.48 0.16 2194 0.53 0.21 1227 0.51 0.17

Total 16581 0.43 0.08 2023 0.49 0.14 1114 0.44 0.09Note: An ANOVA analysis suggested that the sexual orientation was significantly associated with experiences and perceptions on school environment. The LGB students and those with a uncertain orientation were most likely to have negative experiences and perceptions. Students who did not answer the question of sexual orientation were also more likely to report negative experiences and perceptions than those heterosexual students. The associations were significant at 0.001 level (p value) for all dimensions.


Student Perceptions by Sexual Orientation Relative to Heterosexual (=0.0) n~18,000


Violence Index by S3 School


Perceived School Safety by S3 School


Sample School Report (1)


Sample School Report (2)


What other analyses would help you?

Contact us:

Paul Moberg: [email protected]

(608) 263-1304

Daphne Kuo: [email protected]

(608) 261-1154

University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice...

Documents

Transcript of University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice...