SOUTH DAKOTA’S SFP SIG EVALUATION Population Health Institute - University of Wisconsin - Madison...

SOUTH DAKOTA’S SFP SIG EVALUATION

Population Health Institute - University of Wisconsin - Madison

1

Doug Piper, Senior ScientistPopulation Health Institute

School of Medicine and Public HealthUniversity of Wisconsin – Madison

SPF SIG Advisory Council MeetingChamberlain, South Dakota

September 20, 2010

Intention of the SPF SIG Project


2

The SPF SIG Project was designed by the folks at CSAP to change the way substance abuse prevention is done in the states.Emphasizing data driven planning Evidence-based programs Evaluation•Complex•Resource intensive•Largely driven by CSAP requirements


3

Expectations for State and Local

SPF SIG Grantees Fidelity implementing the SPF process at the state, tribal and coalitions level will increase prevention capacity and a build a stronger prevention infrastructure at all levels.

“Infrastructure” is the underlying foundation or basic framework which is necessary for an organization or system to function.

The components of strong prevention infrastructure might include: A unifying theory of prevention;

Effective leadership, Readiness to collaborate organizations and

agencies;


4


SPF SIG Grantees:Components of a strong

Infrastructure (cont.)


5

Increased prevention capacity & stronger prevention infrastructure at the local level will lead to…. • Community-level prevention system changes independent of the direct allocation of SPF SIG funds.• Delivery of evidence-based, culturally appropriate direct service and environmental prevention programs.• Further refinement and efficiency of planning and implementation.


SPF SIG Grantees


6

The delivery of evidence-based programs implemented with fidelity will lead to improvements in community level and individual outcomes. Integration of cultural competency principals and practices at the state and local levels will promote sustained prevention capacity and infrastructure development.


SPF SIG Grantees

Population Health Institute - University of Wisconsin - Madison 7

South Dakota’s AOD Prevention

System

Funding sourceSD governmental unitProgramsState-wide advisory bodyMOUs with DADA to run programs


8

South Dakota SPF SIG Goals for the Local

CoalitionsThe South Dakota SPF SIG Advisory Committee, based on extensive epidemiological data, have chosen the following goals:

1.Reduce the incidence and consequences of underage drinking in South Dakota among youth ages 12 to 20 years.

2.Reduce the incidence and consequences of binge drinking among 18 to 25 year olds in South Dakota.


9

South Dakota’s 30 day use for alcohol and binge drinking for 18 to 25 year olds

compared to National data

NSDUH data


10

Primary Evaluation Objectives

1.Describe the process through which the SPF project is implemented in South Dakota and at the community and tribal levels.2.Measure prevention capacity and infrastructure development in South Dakota and the community and tribal levels.3.Determine to what extent the SPF has produced changes in targeted communities and individual outcomes.


11

Evaluation Types1.Monitoring is checking records and doing site visits to ensure that the intervention included in the contractual work plan is being implemented with reasonable fidelity. This is DADA’s role and not that of the evaluation team.2.Process evaluation documents the program implemented by tracking the input or activities. It defines the independent variable.

3. Outcome evaluation measures the program’s effects that you expect to achieve after the program is implemented

Program actually

implementedInputs Outcomes –

targeted behavior


12

• Attempt to change the surroundings in which people choose to use, or not use, substances. • Aim at changing environmental conditions that contribute to substance abuse. Accessibility of substances, Law enforcement efforts to reduce illegal activity, penalties for illegal activity, as well as Social norms regarding substance use.

• Often involve changing laws, policies, and regulations (LPRs) to make it more difficult for people to abuse substances. • Intended to change individual-level behavior by creating environments that dissuade substance abuse.

Environmental Strategies

13

Logic Models and Evaluation* ENVIRONMENTAL EXAMPLE 1

South Dakota SPF SIG Priorities: (check one)*__Underage drinking among 12 to 20 year olds. __Binge drinking

among 18 to 25 year olds.

Logical connection

Substance-related

consequences

Intervening variables & underlying conditions

StrategiesOutcome evaluation

indicators

Car crashes among young adults

Few officers assigned to enforce alcohol & DUI laws

Work with local law enforcement to increase surveillance at bar time

Law enforcement records show an increase in patrols at bar time

Community norms accept heavy drinking

Media advocacy to increase community concern about drinking and driving

Community survey of 18 to 25 yr olds show a reduction in drinking and driving

Ban alcohol price promotions & happy hours

Passage of new ordinances or agreements to ban or reduce alcohol promotions

* If you are targeting both priorities you must complete a separate Logic Model for each one.

14

Logic Models and Evaluation* NVIRONMENTAL EXAMPLE 2

South Dakota SPF SIG Priorities: (check one)__Underage drinking among 12 to 20 year olds. __Binge drinking

among 18 to 25 year olds.

Logical connection

Substance-related

consequences


StrategiesOutcome

evaluation indicators

Disruptive beer parities among high school students

Parents don’t monitor parties or alcohol supplies

Social event monitoring and enforcement

Existing student survey shows overall reduction in use among 18 to 25

Students arrive at school after having drunk at lunch time

Easy social availability of alcohol (adults provide alcohol)

Stiffen school policies regarding alcohol use among students

Number of alcohol-related school suspensions decreases

Retailers sell to under-aged youth

Enforce under-age retail sales laws

Proportion of negative compliance checks decreases


Individual-focused Prevention


15

This approach to prevention endeavors to change individuals within the community or society. Their surrounding environment is not a target for change to any great degree. They may aim to improve an individual’s refusal skills, increase their connection to the community or promote positive family interaction. Examples are: School based programs Indicated programs which focus on high-risk individuals

16

Logic Models and Evaluation INDIVIDUAL-BASED PROGRAM EXAMPLE

South Dakota SPF SIG Priorities: (check one)*

__Underage drinking among 12 to 20 year olds. __Binge drinking among 18 to 25 year olds.

Logical connection

Substance-related

consequences


StrategiesOutcome evaluation

indicators

Surveys or other epidemiological data show unacceptable rates of underage use

Drinking is considered a rite of passage. “I did it when I was in high school.”

Implement a school-based evidence-based program.

All participants in individually-focused programs must be surveyed with NOMs questions.

Easy social availability of alcohol (adults provide alcohol)

School policies regarding alcohol use among students

Number of alcohol-related school suspensions declines



17

Challenges for Evaluating Community Outcomes of Environmental

Strategies Dearth of data corresponding to the Coalition’s service area and the project’s target populations.Sensitivity of the data to our interventions, e.g. alcohol-related car crash deaths. Immediacy of the intervention’s affects on the target population. Dosage of the intervention penetrating to the target population. May require additional data collection for 20 to 25 year olds for social norms, etc.

Challenges for Outcome Evaluation of Individual-based

Prevention


18

Need a comparison group to show more than just change. CSAP requires all participants in individual-focused programs to be surveyed:• Must include NOMs questions• Data collection methodology must be up to standards• Periodicity of the surveys must be approximately the same. Local Coalitions may be able to secure permission from local school districts to use their data. Only relevant to the underaged drinking target population of 12 to 18 year olds.

Evaluation Tasks for the Coalitions (1 of 3)


19

PROCESS EVALUATION

Phase

1

Phase

2Timeline

Data collected by CoalitionsCommunity-Level Instrument (pt 1) – Community Partner Activities – Coalitions report on CSAPs MRT web-based reporting system

X XAnnually – November 1st

Community-Level Instrument (pt 2) – Intervention Strategies – Coalitions report on CSAPs MRT web-based reporting system

XTwice a year – Nov. 1st & May 1st

Coalitions report progress on SD’s Prevention Management System (KIT data-base)

X Monthly

Coalitions provide Data Collection Reports of what data was collected, how it was collected and by whom.

XAnnually starting June 1, 2012.

All materials developed by the Coalitions for the SPF SIG project will be forwarded to UW PHI for archives

X XAs appropriate

Coalitions provide data as respondents

Coalition Capacity Survey – Coalitions interviewed by UW Population Health Institute

X X

December 2010 & September 2015



20

OUTCOME EVALUATION

Phase

1

Phase

2Timeline

Data collected by coalitions on individually-focused interventions

Surveys of all participants in individually-focus interventions with the Participant-Level Instrument (PLI) to be developed by UW PHI (including NOMs).

X

Pretest and post-test of all participants

Data collected by coalitions on environmentally-focused interventions

Existing epidemiological data appropriate to intervention,# killed in alcohol-related auto crashes% of auto crashes that are alcohol-relatedUnderage drinking arrestsDUI arrests of persons 21 to 25 yrs old.

X

Submit when data is available, at least annually



21

EVALUATION TRAINING AND TA

Phase

1

Phase

2Timeline

Respond to Needs Assessment Survey of evaluation skills of Coalitions conducted by UW PHI

XOctober 2010

Participate in three formal evaluation trainings

XOct. 4 – 6, 2010 plus 2 dates TBD

Participate in quarterly Learning Communities

XStarting in July 2011

Request TA on evaluation issues from UW PHI

X X As needed

Participate in evaluation site visits X Twice a year

Suggestions for recruiting an on-site evaluator for the South

Dakota SPF SIG Project to be a member of the evaluation team,

housed in Pierre and an employee of the University of Wisconsin –

Population Health Institute.


22


23

Doug Piper5901 Research Park Blvd.Madison, WI 53719

[email protected]

SOUTH DAKOTA’S SFP SIG EVALUATION Population Health Institute - University of Wisconsin - Madison...

Documents

Transcript of SOUTH DAKOTA’S SFP SIG EVALUATION Population Health Institute - University of Wisconsin - Madison...