Evaluation of the Fighting Back Initiativeby Kay E. Sherwood

Evaluation of the Fighting Back Initiativeby Kay E. Sherwood

Presented by Maddie Velez

Presented by Maddie Velez

What is Fighting Back?What is Fighting Back?

A community-based drug abuse prevention

program

A community-based drug abuse prevention

program

Why study this Initiative?

Why study this Initiative?

Shows the importance of taking context into evaluations

Raises questions about how community interventions are conceptualized and evaluated

Provides a warning about the manageability of large-scale, comprehensive evaluations.

Shows the importance of taking context into evaluations

Raises questions about how community interventions are conceptualized and evaluated

Provides a warning about the manageability of large-scale, comprehensive evaluations.

Background: Duration and Scale Contribute to Complexity

Background: Duration and Scale Contribute to Complexity◊ A 12 year initiative◊ Original stakeholders differed greatly from the stakeholders involved 12 years later

◊ Reduction of intervention sites went from 15 to 5

◊ First evaluation team replaced after 2 years.

◊ High staff / leadership turnover◊ Original key leader retires◊ Few examples of credible, successful evaluations that truly measured the interventions impact

◊ A 12 year initiative◊ Original stakeholders differed greatly from the stakeholders involved 12 years later

◊ Reduction of intervention sites went from 15 to 5

◊ First evaluation team replaced after 2 years.

◊ High staff / leadership turnover◊ Original key leader retires◊ Few examples of credible, successful evaluations that truly measured the interventions impact

The Foundation Takes on Substance abuse

The Foundation Takes on Substance abuse

◊ Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse.

◊ First grant was made to Vanderbilt University for $26.4 million in 1988

◊ Foundation explores addressing the national problems of substance abuse and dependence

◊ Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse.

◊ First grant was made to Vanderbilt University for $26.4 million in 1988

◊ Foundation explores addressing the national problems of substance abuse and dependence

Continued…Continued…

◊ July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.”

◊ The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.”

◊ Project STAR and ALERT◊ Poly abuse - combination of mental health problems and substance abuse occurring

◊ July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.”

◊ The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.”

◊ Project STAR and ALERT◊ Poly abuse - combination of mental health problems and substance abuse occurring

New Leadership: Kathryn EdmundsonNew Leadership: Kathryn Edmundson◊ New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement?

◊ An element of racism and elitism in the law enforcement

◊ Expected outcomes

◊ New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement?

◊ An element of racism and elitism in the law enforcement

◊ Expected outcomes

Evaluation I: Lost time, Money, and

Credibility

Evaluation I: Lost time, Money, and

Credibility◊ 1990-1994◊ The first evaluation team replaced, 4 years, $4.6 million, and a baseline

◊ Division between stakeholders missed changes

◊ Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data.

◊ 1990-1994◊ The first evaluation team replaced, 4 years, $4.6 million, and a baseline

◊ Division between stakeholders missed changes

◊ Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data.

A 1996 WatershedA 1996 Watershed

◊ Become unified with an emphasis on prevention, early intervention, treatment, and aftercare.

◊ NPO (National Program Office) moved to Boston University School of Public Health.

◊ NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom.

◊ Board of Trustees makes a recommendation to give the program Fighting Back more time.

◊ Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect.

◊ Become unified with an emphasis on prevention, early intervention, treatment, and aftercare.

◊ NPO (National Program Office) moved to Boston University School of Public Health.

◊ NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom.

◊ Board of Trustees makes a recommendation to give the program Fighting Back more time.

◊ Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect.

A National Program Office Change

A National Program Office Change

◊ Fighting Back reduces # of sites eligible for new funding.

◊ Measure most substance abuse within the communities to be able to do something measurable at community level.

◊ Increasing treatment and treatment capacity an important goal.

◊ Fighting Back reduces # of sites eligible for new funding.

◊ Measure most substance abuse within the communities to be able to do something measurable at community level.

◊ Increasing treatment and treatment capacity an important goal.

1994-2000 Evaluation II

1994-2000 Evaluation II

◊ Consensus 2nd evaluation team does an credible job with difficult circumstances.

◊ 1st Evaluators spend $4.6 million dollars with little to show for it.

◊ Fighting Back Program and evaluation staff is moving forward w/out replacement dollars.

◊ Consensus 2nd evaluation team does an credible job with difficult circumstances.

◊ 1st Evaluators spend $4.6 million dollars with little to show for it.

◊ Fighting Back Program and evaluation staff is moving forward w/out replacement dollars.

Relying on Survey DataRelying on Survey Data

◊ Phone surveys throughout the community.◊ Management Information Systems (MIS).◊ Ethnographic Studies.◊ Community Indicators◊ Four Research questions were identified by the 2nd evaluation team.

◊ Strong correlations between strategies and outcomes.

◊ Community Indicators◊ School survey data difficult to use.

◊ Phone surveys throughout the community.◊ Management Information Systems (MIS).◊ Ethnographic Studies.◊ Community Indicators◊ Four Research questions were identified by the 2nd evaluation team.

◊ Strong correlations between strategies and outcomes.

◊ Community Indicators◊ School survey data difficult to use.

The Price of Relying on Survey Data

The Price of Relying on Survey Data

◊ 1996 residue of distrust◊ Saxe’s research team became known as the “national evaluation

◊ Community has been seen as the “human subject”

◊ National evaluation offer no alternative to outcomes perspective

◊ High emotions surrounding analysis emerged accusations

◊ Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions

◊ 1996 residue of distrust◊ Saxe’s research team became known as the “national evaluation

◊ Community has been seen as the “human subject”

◊ National evaluation offer no alternative to outcomes perspective

◊ High emotions surrounding analysis emerged accusations

◊ Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions

The Evaluation’s Ability to Explain

The Evaluation’s Ability to Explain

◊ Evaluation illustrate all central problems for evaluation

◊ Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it

◊ Fighting Back site activities revised after an initial publication in 1997

◊ Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals

◊ Evaluation illustrate all central problems for evaluation

◊ Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it

◊ Fighting Back site activities revised after an initial publication in 1997

◊ Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals

Measuring and Interpreting Outcomes

Measuring and Interpreting Outcomes

◊ Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data

◊ 3 waves of surveys- 1995, 1997, 1999

◊ Jellinek described early thinking on the evaluation

◊ Presentation of Results - A second area of disagreement

◊ Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data

◊ 3 waves of surveys- 1995, 1997, 1999

◊ Jellinek described early thinking on the evaluation

◊ Presentation of Results - A second area of disagreement

The Continuing Debate and the Foundation’s

Takeaway

The Continuing Debate and the Foundation’s

Takeaway◊ Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04

◊ Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes

◊ Teams were formed ◊ Substance abuse- D.A.R.E. and treatment reform

◊ Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04

◊ Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes

◊ Teams were formed ◊ Substance abuse- D.A.R.E. and treatment reform