Implementing Evidence-Based Practice:

An Overview of the BEST Training with Practitioner

Responses

Presented by:

Sarah E. Bledsoe, Ph.D., M.Phil., M.S.W.School of Social Work

University of North Carolina at Chapel HillJennifer L. Bellamy, Ph.D., L.M.S.W.

George Warren Brown School of Social WorkWashington University

Implementing Evidence-Based Practice:An Overview of the BEST Training

with Practitioner Responses

Sarah E. Bledsoe1

Jennifer L. Bellamy2

Lin Fang3

Jennifer I. Manuel3

Edward J. Mullen3

for the BEST Team

1School of Social Work, University of North Carolina at Chapel Hill2George Warrant Brown School of Social Work, Washington University

3Columbia University School of Social Work

Supported in part by the Willma & Albert Musher Program at Columbia University

Introduction Underutilization of scientific knowledge

Evidence-practice gap

2 Challenges:

Lack of knowledge and training in EBP & ESI

Discrepancies in the definition of EBP & ESI

Consequences:

Practice outcomes

Translational research and knowledge generation

BEST Project Bringing Evidence for Social Work Training

Multi-phase exploratory study

Training-based collaborative agency-university partnership

3 social agencies

Response to:

Limited EBP/ESI implementation research

Push for EBP/ESI in practice and education

From: Shlonsky, A., & Gibbs, L. (2004). Will the real evidence-based practice please stand up? Teaching the process of evidence-based practice to the helping professions. Brief Treatment and Crisis Intervention, 4, 137-153.

Evidence-based practice is the integration of best research evidence with clinical expertise & patient values

Empirically Supported Interventions

Interventions for which there is consistent scientific evidence showing that they improve client outcome

Interpersonal psychotherapy for depression

Therapuetic foster care

Assertive community treatment teams

(Drake, Goldman, Leff, et al., 2001).

EBP in the BEST Training Conceptualized as a professional model

of practice:1. Motivation2. Question development3. Search for relevant research4. Research appraisal5. Application6. Evaluation7. Dissemination

(Gibbs, 2003)

BEST Training

BEST Training 10 training modules

Designed to:

Increase knowledge of EBP

Improve EBP research skills

Foster positive attitudes toward EBP

Identify factors that promote the use of EBP in social service organizations

Overview of the BEST Training with Practitioner Responses

Today we report:

10 training modules

Agency practitioner perceptions of the training

Trainer experiences of implementation

Methods Convenience sample of 3 agencies (n=16)

Process notes and post-training focus groups Tape recorded and transcribed verbatim Interviewer notes of emerging themes NVivo Pre-identified and emergent themes

Data analytic strategies: Triangulation of data sources Multiple coders Member checking

SampleAge 34.9 years (mean)

Female 87.5%

White/Caucasian 37.5%

Asian 37.5%

35K-50K 56.3%

Masters Degree 50.0%

Social Work 81.3%

Social Work License 50.0%

Time at current agency 6.3 years (mean)

Intervention Cooperative agency-university partnership

Earliest stages included partner agencies

Pretraining meetings and focus groups

Guided by agency teams & administrators

Practice-relevant topic selected by agency team

Hands-on training using problem-based learning strategy

Research team provided technical assistance & guided the group process

Modules of the BEST Training1. EBP Introduction and Overview2. Question Selection3. Overview of Research Evidence4. Search Tools5. Search Demonstration6. Troubleshooting the Search7. Evaluating the Evidence8. General Findings and Observations9. Synthesizing Evidence Found10. Action Plan

Module One:EBP Introduction and OverviewI. Definition of EBP

a. Brief history of EBPb. Major motivations to use EBPc. Brief overview of different modelsd. Highlight Gibbs (2003) model: Steps as a general

guidee. What we mean when we say “evidence”

-research evidence-practitioner experience-client preference-agency mission and values-ethics

f.  Disconfirming and confirming evidence g. Feedback from the group/questions/any tweaking

necessary?

Practitioners Reactions Module One“I think we have a fuller view of EBP…[the training] definitely deepened our understanding about the process and let us have a chance to try it out.”

“It feels much more tailored to where we’re coming from and what our interests and values are and not like there’s going to be a protocol that is going to be put onto our practice that we would have to somehow manage to squeeze ourselves into and that wouldn’t be desirable.”

Module Two:Question SelectionI. Question selection

a. List project focus area for the questionb. Discuss elements of “researchable” questions

-Relevance to practice-Not something you already have the answer to-Specific

c. Practice making up a few questions d. Discussion, refinement & selection of question

II. Background Information for the Questiona. Why this question in particular?b. Population

-Age, race/ethnicity-Resources-Key mental health/substance abuse issues

c. Agency -Referral or recruitment process-Resources available -Current available services

Practitioners Reactions Module Two“When you start to research [the practice problem], it can become so overwhelming and this is a way to narrow it so that you get what you want, so that it’s more practical. Its a little bit more time consuming and taxing, but on the other hand, its a time saver, more effective.”

“Learning under the discipline of coming up with a question, we struggled with that for weeks…If we ask about it one way, we’re asking about effectiveness. We ask it another way, we could be asking a comparison question. So all the ways there are to ask a question and the importance of how you ask [is related] to what you get.”

Module Three:Overview of Research EvidenceI. Defining different sources of research

evidence

a. Review of different types of “evidence” from module two

b. Review of different types of research evidence: Outcome studies, Meta Analyses, Program Evaluations, etc.

Practitioners Reactions Module Three“There are certain more theoretical readings that aren’t particularly related to empirical demonstrations of different techniques…and I know they have a place in this…I didn’t emerge with a lot of clarity about where they fit in to the process, but sometimes I think it would be useful for us to dedicate some time to thinking about [theory].”

Module Four:Search ToolsI. Search Engines that are Electronically Available

a. Types of search engines & overlap between themb. Starting from meta-analyses & systematic reviews to more specifics

II. Using Search Termsa. Example of search termsb. Group brainstorm about search terms for questionc. Group search “plan”

III. Using MOLES(Methodology Orienting Locators for Evidence Search)

Practitioners Reactions Module FourIn reference to handouts listing available search engines identified as free or fee based:“That is awesome, that’s great. If we got nothing at all but that, this would have been a bonus.”

“We found that the tools you gave us, to streamline the search…they were great. My problem is, really, not having the time. It’s a really big problem. And also, some of the engines you gave us, we’re not going to be able to access because we would have to pay and it would be nice to have access to everything.”

“I need more time to use the tools because I don’t know how to. I need to actually be practicing. I don’t have a real grasp.”

Module Five:Search DemonstrationI. Electronic Demonstration of Search Using a Database

a. Getting to a database b. Starting a search c. Keeping track of searched termsd. Using abstracts

II. Introduction to Other Potentially Helpful Websites

III. Review of Search Plana. How does the agency see their ability to search?b. Will there be a single person who is a “searcher”?

Several people? Will everyone search? Is there no good candidate?

c. Search worksheet

Practitioners Reactions Module Five“The organization of [the search] helped…using a table to organize the [search]…I think [providing tools] like these would make people more willing to do this.”

“It was really good to have a concrete example so we could do an actual practice run through the process. Just to do it, try out our own key words in session on our computers. I thought that was extremely helpful.”

“I think the major issue probably is access to the actual literature…that’s the only real difficult part. So, other than that, the process is fine.”

“I don’t think people are going to take the time to search for a list of sites and to search for different key words.”

Module Six:Trouble Shooting the SearchI.   Review of Search Experience

a. Amount of material found

b. Where material was found

c. What was easy

d. What was difficult

e. Address general questions and issues

Practitioners Reactions Module Six“To base practice on the evidence that we were [able to access] for this question, I mean we were struggling. We were struggling.”

“When I come to articles on certain populations…I’m not confident enough to use [them on my own] because there aren’t people available for consultation.”

Module Seven:Evaluating the Evidence

I.   Examining Evidence Founda. Review of different types of evidenceb. Example of quality reviewc. Examples of quality rating scalesd. Identify the types of research evidence found by team memberse. Value of meta-analyses & systematic reviews

II.  Evidence Qualitya. Review of the different philosophies/ratings of evidenceb. Match of evidence/research type with question- the same

type of research is not appropriate for all questionsc. General indicators of qualityd. Ratings worksheete. In vivo practice using worksheet via discussion of evidence found

III. Refinement of Search Plan Based on Findings To-Datea. Any search engines not covered?b. Any types of evidence missing?c. Continue collecting and assigning evidence

Practitioners Reactions Module Seven“I think the tools were wonderful, but extremely rigorous and I think we would probably do it a little more impressionistically…I think if practitioners could read [evidence] together, almost like a book review, talk about it, that would feel almost in sync with how we approach issues and challenges.”

“I think it’s really important if you actually were able to wrap your head around the math…I still don’t know which study’s data is really that clean…or if the methods are really that good…having a [statistic] helps you decide that but I can’t see myself going through 15 articles and calculating [a statistic].”

“I can find research articles…but I don’t have confidence interpreting the findings.”

“I start to zone out.”

Practitioners Reactions Module Seven“One question is, how much research is out there that is relevant?”

“Sometimes I have difficulty [evaluating evidence] because a lot of research is done with…other populations and…I have to think, can I really generalize these techniques and would it be applicable to [the] specific population I’m working with? A lot of times I find [the empirical evidence] seems really good but I wonder if it’s helpful or good [for my clients]. This evaluation would be challenging to me.”

“In terms of fit, so it almost would require another piece of training or knowledge around how you judge evidence that’s been produced out of a completely different population. In this case, we had difficulty finding research dedicated to the population that looks like [our clients].”

Module Eight:General Findings and ObservationsI.  Review of General Findings

a. Overall, does the research answer our question of interest?b. Is there a great deal of research related to our question of interest?c. Which types of research are missing?

II.  Review of Research Founda. Start with research synthesesb. Move to individual research articles and other pieces

III. Research synthesis and summary handoutsa. General overview of the research question or hypothesisb. How recent is the evidence?c. Main findingsd. What is the general quality of the evidence?e. How well does it fit?

Practitioners Reactions Module Eight“I really think the good research can incorporate the practice perspective.”

“I just want to say, I did get a sinking feeling when we were…actually evaluating the articles we all found…like, is this the [best available evidence]?”

Module Nine:Synthesizing the Evidence FoundI.   Research Synthesis

a. Title (Year)b. Source Typec. Major Findingsd. Quality and Fit Notes

II.  Summary Statementa. Body of evidence currently availableb. Summary of findingsc. Recommended plan of action (if any)

III. Refinement of Group Search Plan Based on Findings To-Datea. Any search engines not covered?b. Any types of evidence missing?c. Continue collecting and assigning evidence

Practitioners Reactions Module Nine“What we’ve learned about [EBP]..it’s helping us understand how to find the areas of research that are missing and figure out places where other contributions need to be made.”

“There’s a gap between what has been done on the research level and what’s being done everyday on the frontline. It would be great to break that and its hard to do. I know we came up with some ideas about trying to implement some of this research knowledge in more day to day functions here. I think it would be interesting and helpful to do that.”

Module Ten:Action PlanI. Possible Next Steps

a. Summary Statement can form the basis for:-Letter to administration-Training to share findings-Seek training for an identified intervention-Use or development of new instruments or tools-Resource manual-Procedural changes-Request for additional resources or funds-Research at the agency

b. Team Plan-What will we do with our findings?-Who will carry out which steps?-When will these steps be carried out?

c. EBP Next Steps-How will you plan to continue using EBP?-Who will carry out these steps?-When will these steps be carried out?

Practitioners Reactions Module Ten“We need to incorporate it into our daily practice. Our regular team meetings will focus on [EBP]…and incorporate [EBP] into agency structure and promote that type of agency structure.”

“Training is important so people can read the research and…apply it to the situation. I believe we have the responsibility, the agency has the responsibility. I think it is a really challenging process.”

“As problems come up, as puzzles arise. I mean, I have a plan to use this. I’m just waiting for the time.”

Practitioners Reactions Module Ten“You’re seeing how hard it is to take the little attention that we have, compared to everything else we have to do here, to think this way.”

“We don’t generally have someone like [the research team] to call up and say, ‘We really want to get an article on x or y, could you find 6 things or help us find 6 things?’, ‘What do you think about the key [search terms]?’, or ‘Where would you start?”

“The only drawback is time…But it could never be a drawback to using it. It’s valuable.”

“We’re willing to do what we can.”

Limitations Exploratory study Sample size Purposive sample New York City agencies

RecommendationsParticipants highlighted need for: Additional training/supervision in EBP & ESI Continued consultation & technical support More evidence addressing real world practice

problems & populations Increased technical & statistical skills to evaluate

the evidence Addressing real barriers

Limited time & competing demands Limited funding to support EBP Limited access to sources of empirical evidence Agency/organizational culture

Recommendations for Social Work & Social Work Education Training Programs (classroom and field)

Process of EBP Specific ESI Research Skills Statistical Skills

Research Increase partnerships with practitioners Increase practice relevant research

Continuing Education Library access for Alumni & Field Agencies Quality continuing education (EBP & ESI)

Evidence Based Practice & Policy

Online Resource Training Center

http://www.columbia.edu/cu/musher/Website/Website/index.htm