Examining Strategies to Improve the Use of Research ...€¦ · Preventive Standards practice •...
Transcript of Examining Strategies to Improve the Use of Research ...€¦ · Preventive Standards practice •...
@aypf_tweets
“Examining Strategies to Improve the Use of Research Evidence in Child Welfare”
Monday, October 5, 2015
Innova'on and the Use of Research Evidence in Public Youth-‐Serving
Systems
Lawrence A. Palinkas, PhD Albert G. and Frances Lomas Feldman Professor of Social Policy and
Health (funded by William T. Grant FoundaBon, Grant No. 10648)
Vulnerable youth need evidence-‐based services
• One in five youth in the United States are in need of mental health services (New Freedom
Commission, 2003). • There are currently 415,000 youth in child welfare in the United States today (Admin Child Fam., 2015).
• 50% of these youth need mental health services (Burns et al., 2004).
These youth are not geSng the services they need
• Only 20% of youth who need mental health services get the services they need (Kataoka et al., 2002).
• 90% of publicly-‐funded child welfare, mental health and juvenile jus'ce systems do not use evidence-‐based prac'ces (Hoagwood & Olin, 2002).
• Only half of all children in child welfare receive care consistent with any one na'onal standard and less than 10% receive care consistent with all 10 na'onal standards (Raghavan et al, 2010).
Use of research evidence is cri'cal to adop'on and implementa'on of evidence-‐based prac'ces
• Implementa'on of evidence-‐based prac'ces and policies (EBPPs) involves some degree of use of research evidence (URE). – To iden'fy appropriate EBPPs – To support decision to adopt EBPPs – To support implementa'on of EBPPs
• However, the extent to which URE contributes to EBPP implementa'on is unknown.
Specific Aims of My Research
1. Understand and measure the use of research evidence by decision makers of public youth-‐serving agencies.
2. Prospec'vely determine whether use of research evidence predicts stage of EBPP implementa'on.
The CAL-‐OH Study
• Objec've: Determine whether Community Development Teams (CDTs) are more effec've than services as usual in “scaling up” implementa'on of Mul'dimensional Treatment Foster Care.
• Design: Adap've or rolling RCT in which 40 California and 11 Ohio coun'es are randomized into two condi'ons (CDT vs SU).
Methods – Semi-‐structured interviews and focus groups to assess how systems
leaders determine a prac'ce is evidence-‐based and how they acquire informa'on, evaluate it, and apply it in making decisions about adop'ng and implemen'ng new programs and prac'ces.
– Development of two new measures
• Structured Interview for Evidence Use (SIEU) • Cultural Exchange Inventory (CEI)
– Web-‐based survey of 202 leaders of state and county child welfare, juvenile jus'ce and mental health systems.
LESSONS LEARNED
1. Networks within and across agencies affect the flow of informa'on and what research is
ul'mately applied • Leaders of child serving systems
rou'nely create and u'lize networks of other systems leaders for informa'on and advice and to pool resources when implemen'ng EBPs.
• The larger the network with higher levels of in-‐person centrality, the more likely these systems will implement an EBP.
2. Networks can be developed and strengthened through implementa'on strategies like
Community Development Teams
CDT Control
3. Use of research evidence does inform policy and prac'ce
SIEU measures SIC Stage Dura5on Propor5on of ac5vi5es Phase 1 Phase 2 Phase 3
r r r r r Input .15* .03 -‐.10 .29* .37* Process .19 .28 .07 .28 .14 Output .29* -‐.11 .15 .25 .16 Total SIEU score .29* .13 .02 .38** .31*
* p < 0.05; ** p < 0.01
• Engagement in evidence use was significantly associated with stage of implementa'on and comple'on of ac'vi'es during the implementa'on and sustainment phases of the project.
• This, in turn, was associated with significantly more clients being served and a significantly greater likelihood of achieving competency in use of MTFC.
4. Priority is given with respect to how evidence is accessed, evaluated and applied
• Above all, they were most
likely to use the evidence to support exis5ng decisions than to make new decisions.
• Child welfare leaders were more likely than leaders of other systems to ignore the evidence.
Factor Mean SIEU score
Acquisi'on of evidence (Input) 2.84
Acquisi'on from experts 3.13
Acquisi'on from documents and published materials
2.98
Acquisi'on from network members 2.53
Evalua'on of Evidence (Process) 3.74
Self assessment of relevance 4.08
Self assessment of validity and reliability 3.70
Assessment by others 3.40
Applica'on of Evidence (Output) 3.22
Use the evidence to make or support decisions 3.65
Not ignore the evidence 3.18
Total Use of Research Evidence 3.38
5. Systems leaders use three other types of evidence when considering whether to seek and apply research evidence in making decisions
• Evidence of resources necessary and available for making use of
research evidence (supply), • Evidence of the need for research evidence, usually obtained
from local condi'ons of client and service needs (demand), and • Evidence gained from personal experience (i.e., is the research
evidence consistent with prac5ce experience or personal observa5on).
6. Certain ins'tu'onal incen'ves support the use of research evidence
• Accountability – Systems are increasingly being required to u'lize EBPs if they wish to be paid for the services they provide.
– Choice of EBPs requires use of research evidence. • Con'nuous quality improvement
– Ac've engagement in URE can help to insure that current prac'ces meet highest standards and if not, what prac'ces can achieve these standards and how these standards can be implemented and sustained.
7. Two specific obstacles to use of research evidence. #1: The disconnect between “local” and “global”
evidence Global Evidence
• External – originates outside of agency or jurisdic'on
• Based on standards for scien'fic rigor (e.g., RCTs)
• Emphasis on generalizability or transferability or findings from one state or county to another
Local Evidence • Internal – originates within agency
or jurisdic'on, may include administra've data
• Based on personal experience (either involvement in data collec'on and analysis or familiarity with popula'on studied)
• Emphasis on uniqueness of popula'on and its needs (specificity)
7. Two specific obstacles to use of research evidence. #2: The disconnect between the producers and
consumers of research evidence • Evidence constructed on the basis of Randomized Controlled
Trials has poor external validity because these trials omen cannot mirror real world condi'ons.
• Much evidence is produced in the absence of engagement with consumers beforehand, resul'ng in limited u'lity and relevance to local condi'ons. – This is why EBPs are omen used without fidelity, when they are used
at all.
7. Two specific obstacles to use of research evidence. #2: The disconnect between the producers and
consumers of research evidence
• Consequently, the methods we use to generate evidence are omen not properly aligned with the nature of current policy problems. – When you need answers now, wai'ng for a researcher to submit a proposal, get funding to conduct a study, publish the evidence, seek confirma'on of the findings, etc., is just a frustra'ng experience.
8. Research evidence can bener inform policy in 'mely and useful ways
• Through research-‐policy partnerships. • Through the use of local as well as global evidence. • Through the development of bener methods for producing and dissemina'ng research.
9. Effec've partnerships in child services possess a set of common elements
• An effec've and sustainable research-‐prac'ce partnership builds upon the exis'ng organiza'onal cultures of research and policy/prac'ce.
• However, it is not merely an aggrega'on of these cultures but rather the product of their transforma'on resul'ng from the exchange of understandings, values, aStudes, and rules for engagement that occur between researchers, prac''oners and policymakers.
Intrapersonal (trust, commitment, openness, honesty, willingness to learn, sensi'vity, flexibilty)
Interpersonal (trust, communica'on)
Organiza'onal (roles, distribu'on of funds,
goals, transparency)
Environmental (funding, resources,
demand)
Cultural (guidelines, brokers,
exchanges)
10. Building such partnerships and promo'ng use of research evidence requires your help
• Funding • Engagement in using evidence • Par'cipa'on in partnerships
THANK YOU
QUESTIONS?
Co-Creating the Infrastructure for
Implementing Evidence-Based Practice
American Youth Policy Forum
Washington, DC
October 5, 2015
Allison Metz, Ph.D., Director National Implementation Research Network
Senior Scientist, FPG Child Development Institute University of North Carolina, Chapel Hill
Socially Significant Outcomes
Effective Interventions
Effective Implementation
Methods
Enabling Contexts
Formula for Success
Formula for Success
• Effective collaboration of cross-sector child welfare stakeholders
• Aligned policy and practice • Organizational and system structures
to support and sustain improvement efforts
• Intensive and structured feedback loops
• A commitment to learning and intentional improvement
Enabling Contexts
Challenges and Opportunities
Research evidence can be continually improved, boosting sustainment in practice, and can enable ongoing learning among developers, interventionists, researchers and consumers (Chambers et al., 2014)
Improving Evidence
Improve outcomes: prevent placement, reduce repeat maltreatment, improve family functioning and child wellbeing. Increasing emphasis in child welfare on data-driven decision-making. 11 evidence-based models serving 3,000 families at any point in time.
Case Example NYC Public Child Welfare System
Making It Happen Stakeholder Interactions
Research demonstrates that successful uptake of evidence requires genuine interaction among stakeholders. However, we know that in many instances, collaborations among stakeholders, including researchers and community members, are strained by a lack of mutual understanding of each other’s goals and expectations.
Community and Consumers
Intermediary Organizations
Program Developers and Researchers
Service Providers
Funders and Policy-Makers
Co-Creation Environment
Infrastructure Outcomes
© 2014, Metz & Bartley
Making It Happen Co-Creation Arenas
Interactions take the shape of “mutual consultations” that mediate the use of research evidence in complex child welfare service systems and political contexts. • Model Developers and ACS (public agency)
• ACS and Service Providers
• Model Developers, ACS and Service Providers
Making It Happen Mutual Consultation Major
Two-way communication Minor
One-way communication Absent
Lack of communication
joint activities common understanding; communication, negotiation and mutual understanding; development and use of products and processes
one-way communication; lack of common understanding; convincing one another of perspective
disagreement and mutual misapprehension
Ø Intensity Ø Structure Ø Product development
Making It Happen Mutual Consultation
ACS and Model Developers
Address
misalignment between
policies and standards and
practice expectations
ACS and Service
Providers
Build capacity to support the
use of evidence
ACS, Model Developers and
Service Providers
Translate outside evidence into child welfare
context and use data to drive
decision-making and optimization
Co-Creation
Products and Processes
• translate evidence into child welfare context Logic Models
• revised to address misalignment between policy and practice Preventive Standards
• web-based decision-making tool for referrals that uses predictive analytics Referral Systems
• common constructs across models Fidelity Desk Guide
• implementation science to build capacity of agencies to use evidence Learning Modules
• risk assessment, case management Tailor Models
Co-‐Crea5ng Capacity
ACS
Providers Model Developers
Families & Communi5es
Product
Co-Creation Recommendations
Intensive interactions among public child welfare agencies, researchers, and service providers…
with
Highly structured communication and feedback protocols
and a
Focus on product development
Goals for Today’s Session Thank You
Casey Family Programs
William T. Grant Foundation
Goals for Today’s Session For More Information
Frank Porter Graham Child Development Institute University of North Carolina
Chapel Hill, NC
http://nirn.fpg.unc.edu/ www.scalingup.org
www.implementationconference.org
Allison Metz
From Research to Learning, Policy, and Breakthroughs
Alex Morales, CEO Children’s Bureau of Southern California
American Youth Policy Forum: Research Use In Policymaking
October 5, 2015
Key Points to Deepen Learning into Prac'ce, Policy and Breakthroughs
• Launched from a prac'ce-‐based research project to organiza'onal learning culture
• Research that immediately benefits prac''oners • The Disrup've Ques'on: How can society move to scale with limited financial resources?
• Power of integra'ng prac'ce, research, learning, policy and advocacy
• Cri'cal funding for research/evalua'on/learning
30 Years Ago Launched First Prac'ce-‐Based Research Project
• Tes'ng of an interven'on: in-‐home counseling to prevent foster care placement.
• Development of the Family Assessment somware to assist counselors in family assessment, planning interven'ons, and measuring impact.
• Successfully advocated for legisla'on in California to fund in-‐home family preserva'on counseling.
Strengthening Prac'ce with Family Assessment and Planning Somware
Building a Learning Organiza'on
• ChildStrength: Children’s Bureau’s in-‐house research, evalua'on, and con'nuous learning program.
• Engagement of all stakeholders in learning: – Clients – Staff – Management – Board of Directors – Outside researchers
The Breakthrough: Crea'ng a Learning Network of 70+
Organiza'ons and Engaging Parents/Neighbors
• A voluntary network made up of government, non-‐profits, faith community, universi'es, and businesses to work in a new way.
• Engaging volunteer Belong Neighborhood Ambassadors to create a movement where everyone works to build a neighborhood where we all would want to Belong.
An Example of Learning Our Way Forward:
A Scalable Model to Promote Reading
• Improving daily parent reading to their young children under age 5 years from 43% to 60%.
• Harnessing the power of trained volunteer parents to teach & promote reading to their neighbors.
This is What it Looks Like