Comprehensive Assessment:A Data Driven Decision Making Process

University of South Carolina Student Life

VISION

A caring community united for a safer Carolina.

MISSION

To create a campus-community environment that promotes healthy and safe behaviors among faculty, staff, students, and community organizations.

University of South Carolina Student Life

PURPOSE

To build capacity and to perform ongoing assessment, planning, implementation, and evaluation of substance abuse prevention strategies, and to make recommendations for policy change

GOAL

To contribute to the overall reduction of high risk substance use and its consequences at the University of South Carolina

The SPF process enables campus communities to:

• Plan and develop substance abuse prevention infrastructure

• Implement effective prevention strategies

• Environmental approaches and educational initiatives

• Achieve population level outcomes

A Model for Reducing High Risk Drinking Behaviors on Campus: The Strategic Prevention Framework

EVALUATIONMonitor, evaluate,

sustain, and improve or replace

those that fail

IMPLEMENTATIONImplement

evidence-based Prevention programs &

activities

PLANNINGDevelop a

Comprehensive Strategic Plan

ASSESSMENTProfile population

needs, resources, & readiness to

address needs and gaps CAPACITY

Mobilize and/or build capacity to address needs

Sustainability & Cultural

Competency

Our Coalition’s Assessment ProcessCreated the Data, Assessment, and Evaluation Subcommittee

Review all available data sources

Identify High Risk Priority Areas

Develop logic models

Form subcommittees to address each problem priority area

Set goals and objectives for each priority area

Identify gaps in research, data, and resources

Develop a comprehensive assessment toolkit

Develop assessment protocols and data collection tools

Collect comprehensive data

Identify emerging trends

Review environmental strategies to determine how best to address local conditionsSet goals and develop a strategic action plan for the year

Developed a comprehensive assessment toolkit

Developed assessment protocols and data collection tools

Collected comprehensive data

Identified emerging trends

Reviewed environmental strategies to determine how best to address local conditions

Set goals and developed a strategic action plan for the year

Assessment Timeline 2014-15August 2014: Review logic models and strategic plan

September 2014: Present strategic plan to coalition

October 2014: Implement strategies and monitor

November 2014: Implement strategies and monitor

December 2014: Implement strategies and monitor

January 2015: Implement strategies and monitor

February 2015: Review data for strategic planning session

March 2015: Identify problems, resources, readiness, and priority areasApril 2015: Annual Assessment report due

May 2015: Host Annual Strategic Session

June 2015: Review logic models

July 2015: Develop strategic plans

August 2015: Steering committee approves strategic plan

Developed a comprehensive assessment toolkit

Developed assessment protocols and data collection tools

Collected comprehensive data

Identified emerging trends

Reviewed environmental strategies to determine how best to address local conditions

Set goals and developed a strategic action plan for the year

Data, Assessment and Evaluation Subcommittee

To inform and guide the coalition in problem and asset identification, strategic planning and implementation, and outcome evaluation

Purpose

Goal

Objective

PURPOSE

To conduct ongoing, comprehensive community-campus assessment and process/outcome evaluation

GOAL

To To conduct assessment to determine the extent of: • Risk factors associated with the problem • Protective factors (assets and resource availability)• Community-campus readiness to address the problem and fill needs

and service gaps• To conduct process and outcome evaluation of prevention strategies

OBJECTIVE

Data, Assessment and Evaluation Subcommittee

Tasks of the Subcommittee

• Determine the assessment’s purpose and scope• Define the assessment’s goals and objectives• Select the approach and methods for collecting data• Design and pilot test the collection instruments and procedures• Prepare assessment timelines• Collect the data• Analyze the data• Prepare data in a way that describes the root causes and local

conditions that contribute to identified community problems• Develop a problem statement(s) that our coalition will address on a

community wide, environmental level

Source: The Fenway Institute

Comprehensive Assessment

Three Areas:• Problem Identification

and Prioritization• Resource Identification• Readiness for Change

Comprehensive Assessment: #1: Problem Identification and Prioritization

The subcommittee has identified many indicators that help to explain student substance use and patterns. Specifically we look at:

• prevalence of use• locations of use• drinking patterns• consumptions levels• student subpopulation(s) most affected• types of substances being used/abused• average age of use• environmental factors, such as availability, accessibility, and cultural

norms

Comprehensive Assessment: #1: Problem Identification and Prioritization

• Enterprise Risk Management

• Risk Assessment (probability, severity, velocity)

• Risk Control Assessment• Final Risk Score and

Ranking

Comprehensive Assessment:#2: Resource Identification

Identify past and current organizational and individual resources, and categorize those resources by key community sectors.

Identify future resources needed to achieve our desired outcomes

Ask the following questions: o Who has the knowledge we need? o Who has the time to assist? o Who has the human resources we need? o Who can hold us accountable for outcomes? o Who can provide financial support? o Who can encourage us and recognize our successes?o Who has helped with this issue in the past? o Who has an interest in our outcome? o Who has decision making power?o Who cares about this issue? o Who has other resources like space, food, giveaways, that we may need?

Comprehensive Assessment:#2: Resource Identification

Identify quantitative and qualitative data currently being gathered and analyzed.

• AlcoholEdu: Freshmen Student Survey• CORE: Upperclassmen Student Survey • NCHA• STIR: Indicated Student Population• OSC: Student violations and sanctions related to substance use• USCPD: crime data related to substance use • Health and Wellness Center: health consequences related to substance use• Key Informant Interviews• Focus Groups• Environmental Scanning

Comprehensive Assessment: #3: Readiness for Change

Determine important aspects to consider when assessing readiness to change. Considerations include:

• Examining multi-levels and multi-facets of campus and the surrounding community

• Determining political will for change• Measuring shared resolve/commitment to implement change • Determining if there is a shared belief in our capability to make change• Determining if there is a shared value that change needs to occur• Assessing the campus community’s ability to meet task and resource demands• Concluded that the higher the readiness for change, the more likely change will

be initiated, the greater the effort , persistence, and cooperation members will give

Drinking LocationFall 2013 vs. Fall 2014

Problem Identification and Prioritization 2014-15

1

2

3

4

Priority: Bars and nightclubs

Priority: Off campus residences

Priority: Other high risk substance use

Priority: Community organizing and advocacy

Conduct Strategic Prevention Planning

Steering Committee utilized assessment information to:• Identify and prioritize the problem(s)• Develop logic models to identify root causes and

contributing local factors

Coalition members then formed subcommittees to:• Create SMART Goals• Select the intended outcome(s) and measure(s)• Identify intermediate variables and measures• Identify strategies for changing intermediate

variables

USC Problem Priority : Prevalent Drinking in Off Campus Residences

Root Causes Contributing Local Factors

Perceived Risk

Social Access & Availability

Social Norms

Prevalent Drinking

Off CampusResidences

Problem

Social Hosting: Students over 21 provide alcohol to minors at

parties/get togethers

Social Hosting: Students over 21 are not award of social hosting laws

Property management rules vary and do not get enforced

“College” norm to drink alcohol

Multiple jurisdictions and enforcement varies

Students perceive low risk of harm in own apartment

Students perceive low risk of getting caught in own apartment

Underage drinking is acceptable

Driving after DrinkingSexual AssaultsPublic Nuisance

Property Damage

Negative Consequences

Inconsistent Enforcement

Logic Model: USC Problem Priority : Prevalent Drinking at Football Game Days

Root Causes Contributing Local Factors

Insufficient Laws and Policies

Social Access & Availability

Social Norms

Prevalent, High Risk Drinking

On football Game Days

Problem

Tailgate Environment, social hosting, alcohol available with little

monitoring

Excessive amounts of alcohol and time periods available for drinking

Large crowds vs. enforcement resources

Perception of “no rules, no consequences” on Game Days

Identification of illegal acts difficult to manage in large crowds

Policies on tailgating vary due to mix of university/privately owned

properties

Tailgating restrictions are inadequate

Alcohol central to tailgating culture

Underage DrinkingBinge Drinking

Driving after DrinkingBlack Outs

Alcohol PoisoningInjuries

Public IntoxicationProperty Damage

Negative Consequences

Inconsistent Enforcement

USC Problem Priority : High Risk Drinking in Bars

Root Causes Contributing Local Factors

Promotion

Access & Availability

Social Norms

High Risk Drinking in

Bars

Problem

Inconsistent Service Practices (ID checking, over-service)

Extended Drink Hours

Perceived safety: Safety risks are low with friends

Underage drinking is acceptable

Perceived getting caught: Easy to get in bars using fake ID and drink

underage

Social Networking and Social Media Advertising

Inconsistent enforcement of pricing, drink specials, and happy hour laws.

Drinking to intoxication is acceptable

Binge DrinkingUnderage Drinking

Black OutsAlcohol Poisoning

InjuriesSexual Assaults

Property DamageArrests

Negative Consequences

Bar Sales and Services Subcommittee

Long Term Goal By March 2016, reduce high risk drinking at bars by 10%.

Objectives Reduce access and availability, address alcohol promotion through social media, increase perception of risk (harm and getting caught), and change social norms.

Strategies/Activities Collect data on sales and service practices, promotion and advertising through social media, in order to develop and implement a responsibility toolkit and provide merchant education.

Off Campus Safety Committee

Long Term Goal By March 2016, reduce high risk drinking at off campus properties by 10%.

Objectives Decrease availability and accessibility, increase perception of harm and getting caught, consistently enforce policies and laws, improve referrals, and change social norms.

Strategies/Activities Collect data from off property managers, courtesy officers, and students to develop an incentive package, offer education and programming, and improve collaboration with properties, law enforcement and university.

Other High Risk Substance Use Subcommittee

Long Term Goal By March 2016, reduce high risk drinking at off campus properties by 10%.

Objectives Decrease availability and accessibility, increase perception of harm and getting caught, consistently enforce policies and laws, improve referrals, and change social norms.

Strategies/Activities Collect data from off property managers, courtesy officers, and students to develop an incentive package, offer education and programming, and improve collaboration with properties, law enforcement and university.

Community Organizing and Advocacy Subcommittee

Long Term Goal By March 2016, reduce high risk drinking at off campus properties by 10%.

Objectives Decrease availability and accessibility, increase perception of harm and getting caught, consistently enforce policies and laws, improve referrals, and change social norms.

Strategies/Activities Collect data from off property managers, courtesy officers, and students to develop an incentive package, offer education and programming, and improve collaboration with properties, law enforcement and university.

Game Day Task Force

Long Term Goal By March 2016, reduce prevalence of drinking on game day by 5%.

Objectives Reduce social access and availability, consistently enforce policies and laws, and change social norms.

Strategies Collect data on tailgate policies and student drinking patterns on game day to inform and improve law enforcement and collaborations.

Data, Assessment, and Evaluation Subcommittee

Developed the Comprehensive Assessment Tool Kit, a data collection protocol packet for each subcommittee

Purpose: to assist the subcommittees with conducting assessment specific to their priority area

Assessment Tool Kit included:o brief overview of assessment processo data collection guidelineso variety of data collection methods to considero general timeline template to assist with planningo tools and resources to use for data collection

University of South Carolina Student Life

Subcommittee Action Steps

• Review Assessment Tool Kit• Select most appropriate data collection

methods• Choose or develop data collection tools

and protocols• Plan the data collection process and

timeline• Implement the data collection process

Subcommittees considered the following:What is the focus of our assessment?

What concepts do we need to define to ensure clear understanding? (ie. How do we define “high risk drinking”?)

What methods of data collection are best for our assessment’s focus?

What resources do we need to collect the data? (financial, staff time, training, locations, etc.)

Who will collect the data?

How will we implement the data collection process?

How will we address confidentiality?

Who will our participants be? How will we recruit them?

How many participants do we need for our data collection?

Where will we conduct the assessment?

How much time is needed to administer the data collection process?

How will we measure the data?

How will we ensure the tools we use are culturally appropriate?

How will we pilot/adjust the tools with potential participants?

Review Assessment Toolkit

Consider Collection Methodsand Timelines

Subcommittees considered the following:

• Key Informant Interviews• Focus Groups• Written Surveys• Community Forums• Environmental Scans• Social Media Analysis• Policy Review• Benchmarking

Select or Create Data Collection Tools and Protocols

Data, Assessment and Evaluation Subcommittee developed the following tools:

• Written surveys• Focus group questionnaires and protocols• Key informant interview questionnaires and

protocols• Social Media Collection Tools• Environmental Scan Surveys

• Development of Tools/Instruments to assess Over-service & Effectiveness of Responsibility Toolkit: – Establishment Observation Tool, Volunteer Observer

Protocol, and Establishment Observation Supervisor Checklist;

– Utilization Assessment Tool; and – Patron Survey

Example:Bar Sales and Services Subcommittee

Responsibility Toolkit

• Raise awareness of negative consequences associated with drinking & driving

• Increase perception of risk of getting arrested for DUI• Raise awareness of SC law prohibiting over-service to

intoxicated patrons• Raise awareness of transportation options for intoxicated

patrons • Raise awareness of increased DUI enforcement

Toolkit Objectives: Patron

• Raise awareness of SC law prohibiting over-service to intoxicated patrons

• Raise awareness that law enforcement is tracking place of last drink

• Raise awareness of civil liability of establishment in event of tragedy

• Raise awareness of increased DUI enforcement • Raise awareness of transportation options for intoxicated

patrons • Increase self-efficacy to refuse service to intoxicated patron

Toolkit Objectives: Employees & Management

• Increase implementation of model policies governing alcohol sales

• Increase establishment accountability to prevent driving after drinking

• Increase partnerships between alcohol retail establishments & Task Force

• Increase participation in responsible beverage service training (PREP)

Toolkit Objectives: Management (behavior)

• Patron/staff ratio• Drink specials/promotions• Age verification systems• Inconsistent systems to designate underage patrons vs. 21

and older• Staff drinking with patrons• Patrons requesting non-alcohol drinks ridiculed, hassled• Retailers see over-service as a personal responsibility issue,

not a retailer issue

Observational Scanning Findings

Conclusions from the Assessment

Community Collaboration:• Challenging issue – multiple systems, sectors • Difficulty engaging alcohol retailers in substance use

prevention efforts• “No one size fits all” – community demographicsResponsibility Toolkit:• No good evidence of effectsObservation Tool: • Provided documentation for both (a) pre intervention

baseline and (b) post intervention • Manageable and easy to administer• Observer agreements (reliability) from 70-99%--

EXCELLENT

Implications for Future Coalition Prevention Efforts

• Continue assessment activities (bar sales and services observations, key informant interviews with bartenders and shuttle drivers; frequent, brief student surveys, etc.)

• Continue collaboration efforts, particularly with retail establishments• Establish a student substance abuse cost analysis for key stakeholders• Utilize science concerning key variables and effective interventions• Establish a city-college data monitoring system (MIS)—reflect USC plus

city.• Publish article “Observational Documentation in Evaluating Alcohol Over-

service in Bars and Restaurants” for professional prevention journals to share findings with other college communities

Rhonda DiNovo dinovo@mailbox.sc.edu Phone- (803)-777-3933

Questions & Answers