Training and Technical Assistance Supporting Evidence-based
Prevention and Intervention, and Juvenile Justice System
Improvement September 9, 2014 Evidence-Based Prevention and
Intervention Support Center (EPISCenter)
Slide 2
1.PCCDs Approach to Prevention 2.Implementation Quality &
Monitoring 3.The Role of Technical Assistance Strategic
Coordination PAYS, CTC, EBPs, EBIs Systems for Data Collection
& Reporting Virtual Technical Assistance 4.Outcomes of a State
Agency & University Partnership Presentation Outline 2
Slide 3
1.Using data to identify community risk and needs (CTC)
2.Identifying local services to match those needs (CTC) 3.Assessing
additional programmatic needs; CTC, EBPs, fit & feasibility
4.High fidelity and quality implementation of EBPs 5.Roll-up and
tracking of implementation and outcomes data 6.Estimating
return-on-investment/cost-benefit analysis 7.Supporting
sustainability from seed grants to post-funding 8.Developing
statewide capacity for prevention across all levels Key Aspects and
Activities 3
Slide 4
Support to Community Prevention Coalitions Improve Quality of
Local Innovative Programs and Practices Support to Evidence-based
Prevention & Intervention Programs Multi-Agency Steering
Committee (Justice, Welfare, Education, Health) Intermediary and
State-level Prevention Support System EPISCenter: Initiatives and
Goals Resource Center Mission: To support the proliferation of
quality prevention and intervention programs aimed at promoting
positive youth development and preventing violence, delinquency,
substance abuse and other problem behaviors in children and
adolescents. Implementation Quality Broad-scale Dissemination
Long-term Sustainability 4
Slide 5
PART I PCCDS APPROACH TO PREVENTION: A RESEARCH-INFORMED
STRATEGY
Slide 6
Moving From Prevention Science... Problem Response Define the
Problem Identify Risk & Protective Factors Develop & Test
Interventions Implement & Evaluate Programs 6
Slide 7
We know a great deal about how youth problems develop, and how
to effectively prevent them (& reduce prevalence) Known risk
& protective factors Multiple domains of influence Community,
family, school, peer, individual Multifinality and equifinality A
public health approach to public safety Different trajectories
(early vs. late starters) Criminogenic impact of intervention
Prevention Science Background 7
Slide 8
Programs/services can be placed along a continuum of confidence
based on their evidence or theory How confident are we that this
program or practice is a good use of resources AND improves
outcomes for children and families? Very Confident Evidence-based
This program has been rigorously evaluated and shown to work
Research-based This program is based on sound theory informed by
research Promising Approaches We really think this will work but we
need time to prove it Best Practices Weve done it and we like it
EFFECTIVEHARMFUL Iatrogenic (Harmful) This program has been
rigorously evaluated and shown to be harmful Ineffective This
program has been evaluated and shown to have no positive or
negative effect PromisingIneffective unknown Very Confident
*Bumbarger & Rhoades, 2012 The Continuum of Confidence 8
Slide 9
To Prevention Service Provide Technical Assistance Set &
Collect Performance Measures Monitor Quality of Program
Implementation Assess Public Health Impact Response 9
Slide 10
We also know a great deal about factors that influence quality
of program implementation Champions Stakeholder buy-in Program
credibility Sites capacity for implementation Strong site
coordinators Training in the program model Implementer confidence
in delivery Proactive-ness and responsiveness of technical
assistance Implementation Science Background 10
Slide 11
Prevent dependency, delinquency, and ATOD use to the greatest
degree possible (primary prevention) Intervene effectively with
youth for whom primary prevention is not sufficient Allow
communities the flexibility to select strategies that best meet
local needs Create community-level infrastructure for strategic
prevention planning and coordination Provide accountability and use
scarce resources efficiently ULTIMATELY. Pennsylvanias EBP
Dissemination Model To move the needle on key indicators of
(behavioral) health at the POPULATION level PA Youth Survey + CTC
prev. infrastructure & prioritize RPFs + Targeted support for
selected EBPs + Technical assistance to promote quality,
dissemination, and sustainability = Population level impact 1989
1994 1998/2001 2001/2008 2003/2005 11
Slide 12
Risk Factors (Causes) Adolescent Problem Behaviors (Outcomes)
Equifinality Key Point: There are multiple routes to address a
targeted behavior problem. Multifinality Key Point: When we target
risk factors (the underlying causes of behavior), we address more
than one problem behavior. 12
Slide 13
Risk Factors (Causes) Adolescent Problem Behaviors (Outcomes)
Implications: State and local agencies interested in efficiently
and effectively addressing youth problem behaviors should
collectively focus on underlying risk and protective factors that
drive common and shared problems. 13
Slide 14
Measures risk and protective factors across multiple domains. A
voluntary survey conducted in schools every other year for youth in
6 th, 8 th, 10 th, and 12 th grades. Adapted from the Communities
That Care Youth Survey, additional questions added on gambling,
prescription drug abuse, other anti- social behaviors, and
experience of trauma and grief. All CTC Sites are essentially
required to use it, and many additional schools volunteer to
participate. 2013 PAYS: 200,000+ youth, 335 school districts, 70
other schools The Pennsylvania Youth Survey (PAYS): PAs Essential
Tool for Prevention Planning 14
Slide 15
Creating Fertile Ground for EBPs: The Role of PAYS in
Data-Informed Prevention Planning (The Communities That Care model)
Collect local data on risk and protective factors Use data to
identify priorities Select and implement evidence- based program
that targets those factors Re-assess risk and protective factors
Form local coalition of key stakeholders Leads to community synergy
and focused resource allocation 15
Slide 16
A Review of Research Findings on PA Coalitions Connection to
coalitions increases likelihood of program sustainability 2-yrs
after funding ends. CTC more likely than non-CTC to be sustained.
Connection to CTC coalitions reduces likelihood of making program
adaptations over time (4-yr longitudinal study) Knowledge of the
CTC model significantly predicts coalition providing downstream
support for evidence-based programming. EBP implementers report CTC
assistance in: Mobilizing the community Supporting actual
prevention efforts Promoting evidence-based programming Assisting
in impactevaluation 16
Slide 17
5 year Longitudinal Study of PA Youth % Change of CTC/EBP Youth
Over Comparison Group 419 age-grade cohorts over a 5-year period
Youth in CTC communities with EBPs: Lower rates of delinquency
Greater resistance to negative peer influence Stronger school
engagement Better academic achievement Feinberg, M.E., Jones, D.,
Greenberg, M. T., Osgood, W. D., & Bontempo, D. (2010). Effects
of the Communities that Care model in Pennsylvania on change in
adolescent risk and problem behaviors. Prevention Science, 11,
163-171. Delinquency Negative Peer Influence Academic Performance
School Engagement 17
Slide 18
Impact on Juvenile Court Placement Rates: Comparison of
Placement Rates for Counties* With and Without an EBI Bumbarger, B.
K., Moore, J., & Rhoades, B. (2010). Impact of evidence-based
interventions on delinquency placement rates. Presentation at 2011
Society for Prevention Research annual meeting. Washington, DC.
18
Slide 19
Evidence-based Programs are a wise investment of state
resources Communities with EBPs embedded in the context of CTC have
lower levels of delinquency and youth drug use * EBPs in PA produce
an overall return of $5 for every $1 invested a statewide return
measured in hundreds of millions ** Conservative estimates of the
CTC model demonstrate a $5 return, more realistic estimates
indicate a $10 return*** The Big Picture: * Feinberg, M.E., Jones,
D., Greenberg, M. T., Osgood, W. D., & Bontempo, D. (2010).
Effects of the Communities that Care model in Pennsylvania on
change in adolescent risk and problem behaviors. Prevention
Science, 11, 163-171. ** Jones, D., Bumbarger, B., Greenberg, M.,
Greenwood, P., and Kyler, S. (2008). The Economic Return on PCCDs
Investment in Research-based Programs: A cost-benefit assessment of
delinquency prevention in Pennsylvania. Prevention Research Center,
Penn State University. *** Kuklinski, M. R., Briney, J. S.,
Hawkins, J. D., & Catalano, R. (2012). Cost-benefit analysis of
Communities that Care outcomes at eighth grade. Prevention Science,
13, 150-61. 19
Slide 20
PART II IMPLEMENTATION QUALITY & MONITORING
Slide 21
Deterrent effect with a strong research design Sustained effect
Multiple site replication Basis for Evidence in EBPs 21
Slide 22
When communities tweak a program to suit their own preferences
or circumstances, they wind up with a different program whose
effectiveness is unknown. Blueprints for Healthy Youth Development
www.colorado.edu/cspv/blueprints ? 22 Deviating from Program
Design
Slide 23
Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler,
I. N. (2011). Putting the pieces together: An integrated model of
program implementation. Prevention Science, 12(1), 23-33 The
Connection Between Implementation & Outcomes 23
Slide 24
Unforeseen challenges (time, resources, population access) Lack
understanding of the programs underlying theory Implementers lack
necessary skills Programs that are not user friendly Lack of
administrator support Status quo 24 Barriers to Fidelity
Slide 25
Based in understanding of programs theory of change/logic model
Data collection Process measures (e.g., # of lessons delivered)
Outcome measures (e.g., reduced antisocial attitudes) Implementer
& participant feedback (e.g., how well did Session 6 go?) Use
of valid, reliable survey tools (developmentally appropriate,
actually measures program targets) Observation of program delivery
By developer, or certified trainer/implementer Implementer &
administrator review and reflection on data Adjustments to
implementation made accordingly Elements of Implementation
Monitoring 25
Slide 26
Implementation Monitoring Example 26
Slide 27
PART III THE ROLE OF TECHNICAL ASSISTANCE 27
Slide 28
Support to Community Prevention Coalitions Improve Quality of
Local Innovative Programs and Practices Support to Evidence-based
Prevention & Intervention Programs Multi-Agency Steering
Committee (Justice, Welfare, Education, Health) Intermediary and
State-level Prevention Support System EPISCenter: Initiatives and
Goals Mission: To support the proliferation of quality prevention
and intervention programs aimed at promoting positive youth
development and preventing violence, delinquency, substance abuse
and other problem behaviors in children and adolescents.
Implementation Quality Broad-scale Dissemination Long-term
Sustainability 28
Slide 29
Methods of Technical Assistance Strategic Plan & Outreach
Resource development, distribution Fact sheets, how-to guides,
webpages Trainings In-person, webinars, YouTube videos Networking
meetings (quarterly) CTC & program-specific, cross-over
meetings Connect with, learn from, and problem-solve with peers
On-site consultations and implementation plan development Training,
fidelity, outcome measurement, implementation, connection to
coalition 29
Slide 30
STRATEGIC COORDINATION
Slide 31
Technical Assistance Cross-Systems Engagement Cross-Systems
Engagement Policy Recommendations Research State Agencies
Cross-agency coordination System-level barriers Gap analysis
Program Developer Programmatic barriers to implementation/
sustainability PA-based Trainings Group discounts Implementing Site
Recruitment Data collection Local stakeholder buy-in 31
Slide 32
Strategic Connections 32
Slide 33
Strategic Connections DDAP trainings in CTC needs assessment
training For Single County Authorities to assess local program and
service needs Connecting with Intermediate Units to discuss
programmatic needs and availability of PCCD funding for EBPs
Virtual web-meetings with PA stakeholders to review evidence- based
intervention utilization and outcomes data, including JCJC, OMHSAS,
CJJT&R, JJSES Stage 3 Discussions with developers of
evidence-based programs that may meet PA needs identified by gap
analysis; fit and feasibility EPISCenter prominently featured in
the Prevention Research Centers strategic plan Consultation to
other states on development of state-level infrastructure for
prevention ( PA is a model!) 33
Slide 34
PA Infrastructure for Efficiencies: SFP 10-14 Training &
Quality Assurance Trainer cadre trained to train up SFP
facilitators and observers Improved turn-around time for scheduling
training Decreased PA funds going to out-of-state trainer travel
expenses Quality assurance designees Improved turn-around time for
scheduling QA visit, and for receiving QA feedback and
certification letter IYS Training & Curriculum Discounts
In-state IYS planned training dates Improved ability to plan for
training due to availability Increased access to timely training
Decreased out-of-state travel expense for grantees Curriculum
discounts to PCCD grantees EPIS as ordering & invoicing hub
Scaled discounts according to quantity of curriculum ordered
Decreased PCCD cost for materials Strategic Connections 34
Slide 35
Gap Analysis for PCCD: Phase I: Review state-level PAYS and
Juvenile Justice data to identify risk & needs (PAYS 2011,
2013, Disposition 2012, Recidivism 2007-09). Phase II: Identify
programs that address PA needs, determine level of evidence in
program effectiveness, recommend programs for PA to support (what
remains are true gaps). Phase III: Drill-down to county-level
indicators of risk using PAYS and JJ data, draw in additional data
from other sources (Dept. of Health, PDE, etc.), identify strengths
(positive indicators, coalition presence, prior funding), recommend
programs and strategies to address youth needs. 6 person team
comprised of EPIS representation across CTC, prevention,
intervention, and juvenile justice staff. Phase I and II
presentation and program recommendations are posted online:
http://episcenter.psu.edu/gaps Strategic Development 35
Slide 36
Cross-pollinating across CTC and SPEP: Raising awareness in
community prevention coalitions of the JJSES activities, SPEP SPEP
focus on quality improvement has implications for all youth-serving
programs More widespread use of the YLS will result in coalitions
ability to use that data in conjunction with PAYS to determine
their youths risks and needs Strengthening the role and connection
between local prevention efforts and local juvenile justice efforts
Strategic Development 36
Slide 37
SUPPORT FOR THE PAYS
Slide 38
Presentations & Trainings PAYS 101 & 2013 Webinars PA
Safe Schools Conference Commonwealth Prevention Alliance (CPA) CTC
Regional Meetings Resources Short, online tutorial videos Guide for
interpreting data Templates for sharing community data PDE Guide
& Workbook PAYS 2013 Community/district recruitment Advisory
Group Beyond 101: Advanced topics Using public health model
Prevention planning Grant writing Connecting to
coalitions/community Sustainability Support for Informed Use of the
PAYS 38
Slide 39
Support for Informed Use of the PAYS 39
Slide 40
Support for Informed Use of the PAYS 40
Slide 41
41 Support for Informed Use of the PAYS
Slide 42
SUPPORT FOR CTC
Slide 43
Communities That Care (CTC) Drug-Free Communities (DFC)
Strategic Prevention Framework (SPF) Integrated Services Plan
Hybrid models Support to Community Prevention Coalitions Improve
Quality of Local Innovative Programs and Practices Support to
Evidence-based Prevention & Intervention Programs EPISCenter
Initiative Areas 43
Slide 44
Creating Fertile Ground for EBPs Risk-focused Prevention
Planning (the Communities That Care model) Collect local data on
risk and protective factors Use data to identify priorities Select
and implement evidence- based program that targets those factors
Re-assess risk and protective factors Form local coalition of key
stakeholders Leads to community synergy and focused resource
allocation 44
Slide 45
Support for the Communities That Care Process 45
Slide 46
New grantee orientation Mobilizer training Milestones &
Benchmarks assessment with the Board (grant start, middle, end)
Co-development of implementation plan On-site trainings of
mobilizer, board, and workgroups in CTC process Quarterly
networking meetings (regional NW, SW, C, NE, SE) Quarterly site
visits Monthly phone consults between TA provider and mobilizer The
PCCD CTC Grantee: TA & Milestones 46
Slide 47
Assessing & Supporting Community Coalitions Web-based data
collection from CTC board members Board membership, leadership,
relationships, work style CTC process Programs implemented Barriers
experienced Technical assistance accessed Provide feedback to sites
on coalition functioning Summary report to TA consultant Report
presented to CTC site Used for strategic planning 47
Slide 48
Virtual Technical Assistance: CTC Resources 48
Slide 49
SUPPORT FOR EBPs
Slide 50
ART - Aggression Replacement Training BBBS - Big Brothers Big
Sisters IYS - Incredible Years (Parenting; Basic & Advanced)
IYS Incredible Years (Youth; Classroom & Small Group) LST -
Life Skills Training OBPP Olweus Bullying Prevention Program PATHS
- Promoting Alternative Thinking Strategies PTNDA - Project Toward
No Drug Abuse SFP 10-14 Strengthening Families Program 10-14
Support to Community Prevention Coalitions Improve Quality of Local
Innovative Programs and Practices Support to Evidence-based
Prevention & Intervention Programs EPISCenters Four Focal
Initiatives 50
Slide 51
Functional Family Therapy (FFT) Multisystemic Therapy (MST)
Multi-dimensional Treatment Foster Care (MTFC) Support to Community
Prevention Coalitions Improve Quality of Local Innovative Programs
and Practices Support to Evidence-based Prevention &
Intervention Programs EPISCenters Four Focal Initiatives 51
Slide 52
New grantee orientation Initial site visit by Prevention
Coordinator Co-development of implementation plan Spring and fall
site visits, and site ratings Quarterly networking meetings Quality
assurance visit, rating, and feedback by program developer Near-end
of grant Outcomes Report Ongoing quarterly reporting of process
& outcomes data The PCCD EBP Grantee: TA & Milestones
52
Slide 53
General and program-specific capacity for evidence-based
programming Using data-informed decision making strategies
Understanding the program components, underlying theory Coaching in
implementation fidelity (general, program-specific, lesson- or
component-specific) Identifying participant recruitment and
retention strategies for site coordinators Supporting
implementation and outcomes monitoring; data collection and
evaluation Content and Skill Areas for Technical Assistance 53
Slide 54
General and program-specific capacity for evidence-based
programming Developing sustainability strategies; education and
support in other funding streams, needs-based budget
process/planning, medical assistance funding, program transfer to
local infrastructure (e.g., school) Connecting with other learning
community members Facilitating communication with developers,
researchers Building stakeholder buy-in, meaningful reporting on
outcomes Content and Skill Areas for Technical Assistance 54
Slide 55
Virtual Technical Assistance: EBP Resources 55
Slide 56
SYSTEMS FOR DATA COLLECTION AND OUTCOMES REPORTING
Slide 57
Spreadsheet Tools PCCD-funded prevention programs (ART, BBBS,
IYS, LST, OBPP, PATHS, SFP 10-14, PTNDA) INSPIRE Intervention
Programs (FFT, MST, MTFC) Same goals: Generating process &
outcome data, monitoring Increasing reliability & validity of
data Reducing data reporting burden Increasing usability of data by
variety of stakeholders Standardized Data Collection and Support
57
PART IV OUTCOMES OF STATE AGENCY & UNIVERSITY
PARTNERSHIP
Slide 65
History of Research-Based Prevention in Pennsylvania 1994: Key
state leaders introduce Communities that Care (CTC) in PA
Spearheaded by Pennsylvania Commission on Crime & Delinquency
(PCCD) and Juvenile Court Judges Commission 1994-2002: Initiation
of CTC funding by PCCD 16 cycles of CTC model introduced in ~120
communities 1996: PCCD Co-funding of research for Blueprints
programs 1998: Process Study of CTC conducted by Prevention
Research Center Resulted in creation of statewide TA infrastructure
to support CTC Formalized connection between CTC and EBP Initiative
1998: Initiation of Evidence-based Program Initiative by PCCD 10
cycles of EBPs funded over 13 years, resulting in ~200 EBPs 2001:
Narrowed list of supported EBPs, aka PA Blueprints 2008: PCCD
created Resource Center for Evidence-Based Prevention and
Intervention Programs and Practices Multi-agency Steering Committee
Representing Justice, Welfare, Education, and Health 65
Slide 66
Development and support of communities of practice Including
common public health language in RFAs Statewide surveillance system
(PAYS) Focus on underlying causal mechanisms vs. narrowly defined
behavioral outcomes Community coalitions as local prevention
infrastructure Ongoing monitoring of implementation Requires tools,
skills, and motivation Policy and Practice Innovations Bumbarger,
B. K., & Campbell, E. M. (2011). A state agency-university
partnership for translational research and the dissemination of
evidence-based prevention and intervention. Administration and
Policy in Mental Health and Mental Health Services Research, 39,
(4), p. 268-277. 66
Slide 67
From Lists to Improved Public Health Synthesis and translation
of research to practice, (and practice to research) EBP selection,
dissemination, and uptake Ensuring sufficient implementation
quality and fidelity Understanding adaptation and preventing
program drift Measuring and monitoring implementation and outcomes
Policy, systems, and infrastructure barriers Coordination across
multiple programs and developmental stages Sustainability in the
absence of a prevention infrastructure Bumbarger, B. and Perkins,
D. (2008). After Randomized Trials: Issues related to dissemination
of evidence-based interventions. Journal of Childrens
Services,3(2), 53-61. Bumbarger, B., Perkins, D., and Greenberg, M.
(2009). Taking Effective Prevention to Scale. In B. Doll, W. Pfohl,
& J. Yoon (Eds.) Handbook of Youth Prevention Science. New
York: Routledge. 67
Slide 68
68 From the Field to the Research Journals
Slide 69
Collaborative Policy Innovators: James Anderson Mike Pennington
Linda Rosenberg Keith Snyder Clay Yeager Investigators and Authors:
Brian Bumbarger Mark Greenberg Mark Feinberg Brittany Rhoades Louis
Brown Wayne Osgood Michael Cleveland Damon Jones Jennifer
Sartorious Julia Moore Brendan Gomez Richard Puddy Stephanie
Bradley Elizabeth Campbell The EPISCenter and research described
here are supported by grants from the Pennsylvania Commission on
Crime and Delinquency. Special thanks to the staff of the Office of
Juvenile Justice and Delinquency Prevention (OJJDP). 69