EVIDENCE BASED PROGRAMSDr. Carol Albrecht Utah State Extension

Assessment Specialist carol.albrecht@usu.edu

Evidence Based Programs

Relatively new approach Program decisions are based primarily

on scientific method and scientific evidence

Use best evidence available to make decisions about program

Recognizes that knowledge is provisional and subject to refutation

Evidence Based Programs combines (1) Program Manager’s Expertise, (2) Client Attributes, and (3) Research Findings

Research Findings

Client

Attributes

Program Manage

r’s Expertis

e

EBP

Best Research Evidence Available Can Be Found in Current Literature

Step 1 – Review The Literature “Track down” the evidence- ongoing

process Note Inconsistencies Note research suggesting some

interventions are ineffective

Step 2 – Critique the Literature You Find Examine methods used*

Client Attributes

Take values, concerns, expectations of client into consideration

Involve clients in decision making process

Determine if this program will “fits” your client. Not “one size fits all” situation Consider race, gender, idiosyncratic

characteristics

Program Manager Expertise

Use Common Sense Avoid “fallacies” which include

Overgeneralization - Stereotyping Selective Perception/Observation Ex-post Facto Hypothesizing

Provide alternative explanation after the “study”

Accepting “new” as better Accepting “traditional” as better

How to Complete a Literature Review

Step 1 – Formulate a Question

Step 2 – Search for Evidence

Step 3 – Critically Appraise Relevant Studies

Step 4 – Apply the Literature to Your Question and/or Program

Step 5 – Conduct Your own Research

Step 1 – Formulate a Question What objectives do you want to

accomplish? What intervention is necessary to reach

objectives? Does Intervention A work? Is Intervention A cost effective? Does Intervention A work better than

Intervention B? Does Intervention A work for this type of client? Does intervention A work in this kind of

situation?

Useful Acronym for Formulating Question

Use acronym CIAO Client characteristics Intervention being considered Alternative intervention being considered (if

any) Outcome of intervention (does it work?)

Example

Client CharacteristicsIf children involved in 4-H

Intervention…..personally interact with a mentor for several years…..

Alternative Intervention…..or communicate with a mentor over the internet…..

Outcome…..are they more likely to seek post-high school education?

Step 2 – Search for the Evidence

Helpful “hints” to finding articles Use Google or Google Scholar and type in correct

search terms Combine general terms (e.g., volunteers, evaluation

etc.) With more specific terms (i.e., grades in high school,

crop yield etc.)

Top Down and Bottom Up Searches (Mullen 2006) Top Down – Find Articles/Books that have Compiled the

Literature (i.e., meta-analysis on your topic) Bottom Up – Find a very recently published article and

look at references.

Examples of Resources for “Top Down” Searches

*Campbell collaboration: http://www.campbellcollaboration.org/index.html

**Cochrane Collaboration: http://www.cochrane.org Substance Abuse:http://nrepp.samhsa.gov/htpp:/www.modelprograms.samhsa.gov/index.htm Crisis Intervention and Domestic Violence National Resource: http://www.crisisinterventionnetwork.com Expert Consensus Guidelines series: http://www.psychguides.com National Guidelines Clearinghouse http://www.guidelines.gov National Institute on Drug Abuse: http://www.nida.nih.gov/ Substance Abuse and Mental Health Services Administration:http://www.samhsa.govindex.aspx

*Focuses on Social Welfare, Education and Criminal Justice**Focuses on Health Care Interventions

Step 3 – Critically Appraise Relevant Articles

Emphasis Should be on Methods Used to Collect Empirical Information

How to Complete a Literature Review

Step 1 – Formulate a Question

Step 2 – Search for Evidence

Step 3 – Critically Appraise Relevant Studies

Step 4 – Apply the Literature to Your Question and/or Program

Step 5 – Conduct Your own Research

Step 4 – Apply the Literature to Your Research Question

Which intervention currently has the best empirical support?

Is the situation same/similar to yours? Are the participants in the study the

same or similar to the ones in your program/study?

Step 5 – Conduct Your Own Research

Use Literature Review to help you develop valid and reliable indicators.

Use Literature Review to help you determine your sampling technique and sample size.

Use the Literature Review to help you determine the most appropriate type of data collection.

Use the Literature Review to help you determine ways you can further contribute to the research (i.e., what is missing?)

Evaluate impact of program on your participants

Concerns About Evidence Based Programs

Merely Cost Cutting Have to use the methods “in stock”

Evidence Based May not be Applicable Leads to rigid implementation Your participants may be different

Evidence is in Short Supply – not enough research Real World Obstacles Hinder Implementation of

Programs Time and money are scarce

Devalues Qualitative Research Argument that “Empirical Evidence” is not

obtainable

Suggestions for Appraising Journal Articles in the Literature

Step 1 – Identify Purpose Stated purpose of article

This should appear in first paragraph(s).

Step 2 – Literature Review Is it adequate?

They should have at least 15 references.

Appraising Literature

Step 3 - Identify objectives/outcomes Do they identify and define their

outcomes/objecties It must be clear and succinct. They must be clearly related to the

implementation of program. How do they measure their

concepts/outcomes? Are measurements valid? (face validity) Are measurement reliable (consistent)

Could you repeat study?

Appraising Literature

Step 4 – Appraising Sampling Technique It must be clearly specified?

What is their unit of analysis? Do you know exactly how these units will be

selected? It must be appropriate?

Does the sampling technique “fit” with the data collection technique?

Do they draw valid conclusions based on sampling technique? Note – if they generalize they must have random

sample

Remember the Sampling Techniques

Random Sample – can generalize Simple or Systematic Random Multi-stage Cluster Stratified Random

NonRandom Sample – cannot generalize Theoretical or Purposive Snowball Convenient

Appraising Literature

Step 5 – Research Design/Data Collection Technique It must be clearly specified?

Do you know exactly how they will collect their data?

It must be appropriate Keep in mind the strengths and weaknesses of

the different research designs/data collection techniques

Remember Research Designs or Data Collection Techniques

Data Collection/Research Design Survey or Questionnaire Secondary Experimental

Lab Experiments Quasi – Field Experiment/Trial

Field Research (Qualitative) Participant Observation Content Analysis Intensive Interviews Case Studies

Appraising Literature

Step 6 - Critique Tables/Graphs/Charts Their analysis must be correct?

Did they include descriptive statistics of the sample? Did they include inferential statistics and discuss

level of significance? Are their tables constructed correctly?

They must be interpreted accurately. Does the text accurately describe the data?

They should look professional Do they provide sufficient information?

Do they tell you how they measured the indicators used in the tables?

Appraising Literature

Step 7 – Summary and Conclusions Do they accurately summarize their

research? Do they identify strengths and

weaknesses? Do they acknowledge limitations? Do they make suggestions for future

research?

Easy Steps

1. Identify Purpose 2. Critique Literature Review 3. Identify Success 4. Sample Technique 5. Data Collection Technique/Research

Design 6. Tables and Charts 7. Summary and Conclusions

Contact Information

Dr. Carol Albrecht Assessment Specialist Utah State University 979-777-2421 Carol.albrecht@usu.edu