Ohio Alternative Response Project:Evaluation Findings

Presentation to theOhio Alternative Response

SymposiumMay 13-14, 2010

Tony Loman, Ph.D. and Christine Filonow, MSW

Institute of Applied ResearchSt. Louis, Missouri

www.iarstl.org

CUYAHOGA

FRANK LI N

HAMI LTON

MONTGOME RY

S UMMI T

LUCAS

S TARK

BUTLE R

LORAI N

MAHONI NG

LAK E

TRUMBULL

CLE RMONT

WARRE N

PORTAGE

ME DI NA

GRE E NE

LI CK I NG

CLARK

RI CHLAND

FAI RFI E LD

WOOD

COLUMBI ANA

WAYNE

DE LAWARE

ALLE N

AS HTABULA

MI AMI

TUS CARAWAS

GE AUGA

MUS K I NGUM

E RI E

S CI OTO

J E FFE RS ON

ROS S

HANCOCK

BE LMONT

MARI ON

WAS HI NGTON

LAWRE NCE

ATHE NS

S ANDUS K Y

HURONS E NE CA

K NOX

DARK E

PI CK AWAY

AS HLAND

S HE LBY

CRAWFORD

AUGLAI ZE

LOGAN

PRE BLE

BROWN

FULTON

OTTAWA

ME RCE R

UNI ON

HI GHLAND

GUE RNS E Y

CLI NTON

MADI S ON

DE FI ANCE

WI LLI AMS

HOLME S

CHAMPAI GN

COS HOCTON

PUTNAM

PE RRY

J ACK S ON

HARDI N

MORROW

GALLI A

VAN WE RT

HE NRY

CARROLL

FAYE TTE

HOCK I NG

PI K E

ADAMS

ME I GS

WYANDOT

PAULDI NG

HARRI S ON

MONROE

MORGAN

NOBLE

VI NTON

Pilot Counties

Time Line for Evaluation

2008 2009 2010Jn Fb Mr Ap My Jn Ju Au Sp Oc Nv Dc Jn Fb Mr Ap My Jn Ju Au Sp Oc Nv Dc Jn Fb Mr ApPreparati

on

SACWIS and manual/email methods

Monthly case-specific data from workers

Family Surveys

Data collection Analysis/Report

Cost study

Site visits

Site visits

Site visits

Reception/Analysis of pathway tools (c. 10,000)

Reception/analysis of service plans

Random assignment

Cost study site visitsCommunity &

worker surveys

Community & worker surveys

Pathway Assignment and Random Assignment

TR Traditional Investigations

E Outcome/Impact Analysis

Child Maltreatment

Reports

AR Pathway Assignment

Pool of Reports Eligible for AR

Random Assignment

Experimental Group

Cases offered AR

D

Control Group

Control Cases

Inappropriate for AR

AR-appropriate

AR Family Assessments

B

C

Initial Screening For CPS

Screened out

Accepted Report

(Screened in)

A

The Evaluation

Estimated 51.7 percent of reports determined to be AR-appropriate Final study group included 4,529

families (50.5% Exp. and 49.5% control

Alternative Response: What is Different?

Removing the negative:– No formal victims, perpetrators– No substantiation or indication of

abuse/neglect– No entry of adults into a central registry of

abusers Enhancing the Positive

– Establish child safety (create a safety plan)– Engage the family– Focus on broader family needs– Emphasize family participation in decisions

Changes under AR: Negative Emotional Response of

Families

0% 5% 10% 15% 20% 25% 30% 35%

Angry (0.0110)

Afraid (ns)

Stressed …

Irritated …

Anxious …

Dissatisfied (ns)

Worried …

Confused …

Tense (0.0310)

Negative (ns)

Pessimistic (ns)

Discouraged …

Experimental

Control

Changes under AR: Positive Emotional Response

of Families

0% 5% 10% 15% 20% 25% 30% 35%

Relieved (0.0270)

Hopeful (0.0001)

Satisf ied (ns)

Helped (0.0470)

Pleased (0.0150)

Thankful (0.0020)

Comforted (0.0490)

Reassured (0.0010)

Grateful (0.0090)

Positive (0.0090)

Encouraged (0.0001)

Optimistic (0.0760)

Experimental

Control

Changes under AR: Satisfaction with Worker

49.5%

58.4%

39.2%

33.9%

6.8%

3.7%

4.5%

4.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Control

Experimental

Very Satsified Generally Satisfed Generally Dissatisfied Very Dissatisfied

Changes under AR: Involvement in Decision

Making

41.1%

20.7%

9.0%

5.0%

24.2%

54.3%

15.8%

7.5%

5.1%

17.4%

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0%

A great deal

Somewhat

A little

Not at all

No decisions were made

Experimental

Control

Family Characteristics

Single Parent Families Over two-thirds of families reported incomes of

$15,000 or less compared to 8% for Ohio generally.

Compared with statewide statistics, a higher proportion of family caregivers in the study had less than a high school diploma (31 percent) than in the general population of Ohio (13 percent).

The caregiver in over six in ten families was unemployed.

Only 12 percent were employed full time.

79.0%

40.0%

39.3%

29.9%

25.5%

22.4%

23.1%

16.8%

10.0%

1.6%

80.9%

41.0%

39.1%

29.5%

27.9%

26.3%

23.7%

17.6%

10.4%

1.1%

79.9%

40.4%

39.2%

29.7%

26.6%

24.3%

23.4%

17.2%

10.2%

1.4%

0.0% 10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

Food stamps

School breakfast/lunch

WIC

Child support

Utilities assistance

Social security disability

TANF

Housing assistance

Unemployment benefits

Retirement check

Total

Experimental

Control

Characteristics: Welfare and Other Support

Workers Identified Needs

Families tended to cite basic poverty-related needs. Workers indicated these needs among the families

they encountered but also emphasized problems in family functioning, especially parent-child relationships and parenting skills. Perhaps, consonant with this:– Nearly half of caregivers indicated problems in

children’s behavior—especially uncontrollable and aggressive behavior—and childhood depression and anxiety.

– About one-third of family caregivers surveyed reported that their children had problems in school.

Factors in Service Provision

Family need for services Family engagement Time devoted to cases Funding availability Service/resource availability and

worker knowledge Agency patterns of resource use

Family Reports: Level and Types of Services Received

0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%

Help for a family member with a disabilityAlcohol or drug treatment

Counseling services***Mental health services*

Parenting classesPARENTING, COUNSELING TREATMENT=====

Meetings with other parents Respite care

Assistance in homeChild care or day care

Legal servicesLEGAL, CHILD RELATED, DIRECT …

Employment helpJob training or vocational training

Help getting into educational classesEMPLOYMENT SERVICES===========

Welfare/public assistanceMedical or dental services**

MEDICAL OR WELFARE SERVICES====Appliances or furniture*

Money to pay rent**Housing

Car repair or transportation**Other financial help***Help paying utilities***

Food or clothing for your family**POVERTY-RELATED SERVICES=======

Experimental

Control

* Statistical Trend (p < .10) **Significant (p < .05)*** Significant (p < .01)

Family Reports: Level and Type of Services

Poverty-related services increased: AR workers more often provided referrals for or

helped families to receive food and clothing, help with utilities, other financial help, car repair and transportation, money to pay rent or help in obtaining appliances and furniture.

Experimental families under AR also reported receiving more referrals to traditional counseling and mental health services.

No difference was found in the number of services or the provision of direct services between Caucasian and African-American families under AR.

Family Reports: Satisfaction with Help Offered or

Received

34.4%

36.7%

5.6%

2.6%

20.7%

47.5%

33.7%

3.2%

4.5%

11.1%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Very Satsified

Generally Satisfed

Generally Dissatisfied

Very Dissatisfied

No Services Offered

Experimental (AR Family Assessment)

Control (Traditional Investigation)

Worker Reports of Services Provided

Workers reported providing more services, support and assistance under AR and more information about where services could be found.

Workers indicated that basic poverty-related services were provided significantly more often to experimental families, such as rent payments, housing services, help with basic household needs, emergency food, and transportation.

Worker Reports of Services Provided

Under AR, 46.7 percent of AR workers said they were responsible for directly providing or connecting families to resources and services, while only 26.3 percent of TR workers reported this. Correspondingly, AR workers indicated they provided only “information & referral” for 41.2 percent of the services compared to 59.2 percent for TR workers.

AR workers directly assisted with 83.3 percent of services in the category “help with rent or house payments” compared to 30.0 percent for TR workers.

Most Frequent Types of Services Listed in Service

Plans

County (# plans

reviewed) 1st Frequent Service 2nd Frequent Service 3rd Frequent Service

Clark (72)

Mental health/ counseling

35% Appliances,

furniture, linens 26% Housing assistance 22%

Fairfield (69)

Mental health/ counseling

(in-house worker) 55%

Benefit assistance/ budgeting

(in-house worker) 25%

Education/ employment

services 22%

Franklin (214)

Beds, other household items

26% Settlement house

referral 10% Utility assistance 7%

Greene (104)

Mental health/ counseling

30% Utility assistance 10% Housing assistance 9%

Guernsey (43)

Utility assistance 40% Household items 21% Housing and food

assistance 19%

Licking (54)

Household items 35% Housing assistance 30% Utility assistance 28%

Lucas (107)

Clothing or food voucher

35% Beds/furniture/

appliances 23%

Baby items/ household items

20%

Ross Unable to review

Trumbull (136)

Mental health/ counseling/ therapy

53% Clothing or food

voucher 32% Utility assistance 26%

Tuscarawas (17)

Case-management by PCSA

47% Housing assistance 29% Mental health/

counseling 24%

Family and Worker Reports of Services Provided

Experimental families were also somewhat more likely than control families to indicate that the services received were enough to really help.

According to workers, AR families were also more likely to participate in services than control families.

Changes under AR: Level of Contacts of Workers with

Families

2.6

3.4

0.6

1.4

0.7

3.4

4.6

0.6

2.1

1.3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

How many face to face meetings did

you have with members of the family (p = .003)

How many telephone contacts did you have with members of the family (p = .012)

How many other contacts did you

have with a family member (ns)

How many contacts did you have with others on behalf of this family (p = .008)

How many face to face contacts did

other service providers have with a family member (p

= .02)

Control Experimental

The average number of days until case close was 53.6 for experimental families and 44.7 days for control families.

What Changed?Operational Shifts

Longer assessment period Alternative Response workers provided

post-assessment services Family Service Plans Flexible funding Service partnerships

What Changed?Practice Shifts

Removal of barriers to family engagement Approach is less “incident-driven” Increased communication and trust Allows for more preventative support,

services, and follow-up

““With alternative response, I’m not there With alternative response, I’m not there to gather information to make a decision; to gather information to make a decision; I’m there to help the family come to a I’m there to help the family come to a resolution.”resolution.”

What Changed?Service Shifts

Factors: Family need for services Family engagement Time devoted to cases Funding availability Service/resource availability and

worker knowledge Agency patterns of resource use

Worker and Supervisor Responses

Workers and supervisors that performed work related to AR reported observable adjustments in their approach and practice, indicating that AR was implemented as intended and produced positive changes within the agency.

Workers reported feeling more able to intervene effectively with AR families than with non-AR families.

Reactions of AR families to assistance were seen as more positive by workers than the reactions of other families.

Worker and Supervisor Responses

The majority of staff involved with AR stated that the pilot had affected their approach to families a great deal or in a few important ways.

AR-involved staff saw AR as leading to a more friendly approach to families, more family participation in decisions and case planning, and more cooperation from families in the assessment process.

A strong minority (38.9 percent) of county staff involved with the pilot reported that AR had increased the likelihood that they will remain in the field of child welfare.

Worker and Supervisor Comments

“Alternative response has reminded me how blame driven we could be. A lot of the time, it doesn’t matter whose fault it is, as long as the family is willing to work to function better.”

“I have always tried to address concerns without labeling people, but the traditional approach makes this difficult. Now with the alternative response approach it is much easier to focus on the problems and solutions with the family.”

“Our alternative response team is outstanding. From the start to the close of an AR case, each worker brings an attitude of success which is passed on to the family. The ability to be creative in how families are assisted has been critical.”

Community Responses

Familiarity with AR among stakeholders had increased by the end of the pilot, from 45.3 percent in 2008 to 68.3 percent in 2009.

Attitudes toward AR were highly positive among those who were familiar, although a little less than half of all survey respondents were unsure of their opinion.

Nine out of ten judges or magistrates in the pilot counties reported being at least somewhat familiar with the AR pilot.

Those nine also perceived that AR had the potential to lower the number of cases coming to court to some degree.

Child Safety

Short-term child safety from the time of the original report until final contact with families was examined. Child safety problems were identified by workers in a minority of families, 33.2 percent of control cases and 25.4 percent of experimental cases.

When a child safety problem was identified, no statistically significant difference was found between experimental and control families in the extent of improvement or decline in safety.

There was no evidence that replacement of traditional investigations by AR family assessments reduced the safety of the children.

New Reports of Child Abuse and Neglect

Among families entering the study during the first 360 days, 13.3 percent of control families had a new report compared to 11.2 percent of experimental families. This difference was statistically significant.

A proportional hazards analysis that controlled for levels of past reporting on families also confirmed that experimental families that were served through the AR family assessment pathway had fewer new reports than control families that were approached through a TR investigative assessment.

New Reports of Child Abuse and Neglect

Control

Experimental

New Reports of Child Abuse and Neglect

Racial differences in later accepted reports were also examined.

Although study families as a whole were largely in poverty, African-American families were significantly and substantially more impoverished than Caucasian families. Race was taken as a proxy measure for poverty.

Analyses demonstrated that the major positive effects of AR on new reporting of child maltreatment at this point in tracking families appears to have occurred among African-American families. This was interpreted to mean that AR has its greatest effects among the poorest families in the population.

Subsequent Removal and Placement of Children

Differences in out-of-home placement were also examined in the evaluation.

Within the control group 3.7 percent of children had been removed while 1.8 percent had been removed in the experimental group, a significant difference.

This difference also remained significant in the stronger proportional hazards analysis. AR appeared to reduce the number of child removals and out-of-home placements.

Cost Analysis: Indirect Costs

Indirect costs were calculated using cost allocation data and average time that workers spent with experimental and control families.

AR family assessments averaged $940 per family compared to $732 per family for TR investigations. Reflecting increased worker time with families, AR was more expensive in the immediate term.

For subsequent work, experimental families averaged $145 per family compared to $266 for control families. Total costs for control families averaged about $999 per family compared to $1,084 for experimental families. At this point in the follow-up, experimental families were slightly more expensive ($85 per family) overall in indirect costs than control families.

Cost Analysis: Mean Indirect Cost per

Family

Cost Analysis: Mean Direct Cost per Family

Cost Analysis: Total Direct and Indirect Costs, Initial and

Subsequent to ARControl Experimental

Initial direct costs $ 99 $194

Initial indirect costs $ 732 $ 940

Subsequent direct costs $136 $ 48

Subsequent indirect costs $ 266 $ 143

Total Initial Costs $ 831 $ 1,134

Total Subsequent Costs $402 $191

Total Costs $1,233 $1,325

Quotes from AR Families

“The social worker was fantastic. She did not come to our home with predetermined ideas, but came to conclusions based on our family and our home.”

“I was surprised by how much help was offered. I didn’t know they offered you all that extra help. It was appreciated.”

“The caseworker treated us with respect and made us feel like we mattered and that we had our own voice to speak. We enjoyed our case worker coming and explaining things to us and made us feel wonderful.”

“My caseworker was awesome. She saw I wasn’t a bad mother. I just needed a little help to get back on the right track. And I love her for that.”