Using the Ohio Scales for Assessment and Outcome...

Using the Ohio Scales for Assessment and Outcome Measurement in a

Statewide System of Care

OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION

Oklahoma Systems of Care (OSOC) Expansion

Implications for Evaluation

From 2002-2011, using a combination of state funds and funds from Phase IV and Phase VI SAMHSA grants, Oklahoma’s wraparound program expanded from an initial 6 counties to 56.

This rapid expansion necessitated a state evaluation separate from the national evaluation and longitudinal study.

Not all sites would be included in the national evaluation’s longitudinal study.

Data from the national evaluation would cover a small percentage of youths in OSOC wraparound.

Quality improvement and decision support required 100% data collection from enrolled families.

A scalable solution would incorporate line staff as data collectors, so burden – in time and training – had to be limited.


Lincoln

Cimarron Texas Beaver Harper

Ellis

Woods

Woodward

Major

Alfa

lfa

Dewey

Grant

Garfield

Kingfisher

Logan

Roger Mills

Beckham

Greer Kiowa

Custer

Washita

Blaine

Caddo

Canadian

Grady

Har

mon

Jackson

Tillman Cotton

Comanche

Stephens

Jefferson

Cle

vela

nd

McClain

Oklahoma

Potta

wat

omie

Garvin

Carter

Love

Murray

Marshall

Johnston

Bryan

Pontotoc

Sem

inol

e

Choctaw

McCurtain

Atoka

Pushmataha

Hughes

Coal

Pittsburg Leflore Latimer

Haskell

McIntosh

Muskogee

Okm

ulge

e

Creek

Okfuskee Sequoyah

Wagoner

Che

roke

e

Adair

Kay

Noble

Payne

Pawnee

Osage Was

hing

ton

Now

ata

Rogers

Craig Ottawa

Mayes Del

awar

e

Tuls

a

OSOC Sites 2000-2011

2011

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2000


OSOC 2001-2011 Families Served, Sites and Counties by Fiscal Year

38

156

304

483

648 762

852

1088

1246

1405

1667

3 6 7 9

13

20

26

31 34

40

50

3 7

10

13 20

31

44

49

56

53

56

0

10

20

30

40

50

60

70

0

200

400

600

800

1000

1200

1400

1600

1800

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

# of

Sit

es a

nd C

ount

ies

Fam

ilies

Ser

ved

Fiscal Year

Clients Served Sites in Operation Counties Served


Development of State Evaluation

Psychometric tool originally in use would not scale.

Costs and logistics related to credentialing requirements were prohibitive.

Evaluation committee of evaluators, staff, families and youth developed the state evaluation tool set – based on extracts from longitudinal study public domain instruments – in spring and summer of 2004.

2003/2004 national conferences exposed evaluation team members to the Ohio Scales, in use by other Systems of Care communities.

State evaluation committee chose to build OSOC Assessments around the Ohio Scales in July 2004.


Why the Ohio Scales?

Originally designed as a statewide instrument

Qualities appropriate to statewide implementations Self-administration Parent—Administered at baseline

with follow up at 6 month intervals

Youth—Administered to youth aged 12 and older at baseline with follow up at 6 month intervals

Worker—Administered at baseline with follow up at 3 months and then 6 month intervals from baseline

No credentialing requirement

Nominal cost

Brevity

Simple language and item content

Easily understood scoring, interpretation of results

Good and growing research base

Articles on flash drive.


Instructions:

Please rate the degree to which your child has experienced the following problems in the past 30 days N

ot a

t All

Onc

e or

Tw

ice

Seve

ral

Tim

es

Ofte

n

Mos

t of t

he

Tim

e Al

l of t

he

Tim

e

1. Arguing with others 0 1 2 3 4 5 2. Getting into fights 0 1 2 3 4 5 3. Yelling, swearing, or screaming at others 0 1 2 3 4 5 4. Fits of anger 0 1 2 3 4 5 5. Refusing to do things teachers or parents ask 0 1 2 3 4 5 6. Causing trouble for no reason 0 1 2 3 4 5 7. Using drugs or alcohol 0 1 2 3 4 5 8. Breaking rules or breaking the law (out past curfew, stealing) 0 1 2 3 4 5 9. Skipping school or classes 0 1 2 3 4 5

10. Lying 0 1 2 3 4 5 11. Can’t seem to sit still, having too much energy 0 1 2 3 4 5 12. Hurting self (cutting or scratching self, taking pills) 0 1 2 3 4 5 13. Talking or thinking about death 0 1 2 3 4 5 14. Feeling worthless or useless 0 1 2 3 4 5 15. Feeling lonely and having no friends 0 1 2 3 4 5 16. Feeling anxious or fearful 0 1 2 3 4 5 17. Worrying that something bad is going to happen 0 1 2 3 4 5 18. Feeling sad or depressed 0 1 2 3 4 5 19. Nightmares 0 1 2 3 4 5 20. Eating problems 0 1 2 3 4 5

Ohio Scales: Youth Problem Scale Youth Problem Scale (Copyright © January 2000, Benjamin M. Ogles & Southern Consortium for Children)

Produces a score of 0 - 100.

Externalizing Subscale

Internalizing Subscale

Delinquency Subscale


Ohio Scales: Youth Functioning Scale Youth Functioning Scale (Copyright © January 2000, Benjamin M. Ogles & Southern Consortium for Children)

Instructions:

Please rate the degree to which your child’s problems affect his or her current ability in everyday activities. Consider your child’s current level of functioning. Ex

trem

e Tr

oubl

es

Qui

te a

Few

Tr

oubl

es

Som

e Tr

oubl

es

OK

Doi

ng V

ery

Wel

l

1. Getting along with friends 0 1 2 3 4 2. Getting along with family 0 1 2 3 4 3. Dating or developing relationships with boyfriends or girlfriends 0 1 2 3 4 4. Getting along with adults outside the family (teachers, principal) 0 1 2 3 4 5. Keeping neat and clean, looking good 0 1 2 3 4

6. Caring for health needs and keeping good health habits (taking medicines or brushing teeth) 0 1 2 3 4

7. Controlling emotions and staying out of trouble 0 1 2 3 4 8. Being motivated and finishing projects 0 1 2 3 4 9. Participating in hobbies (baseball cards, coins, stamps, art) 0 1 2 3 4

10. Participating in recreational activities (sports, swimming, bike riding) 0 1 2 3 4 11. Completing household chores (cleaning room, other chores) 0 1 2 3 4 12. Attending school and getting passing grades in school 0 1 2 3 4 13. Learning skills that will be useful for future jobs 0 1 2 3 4 14. Feeling good about self 0 1 2 3 4 15. Thinking clearly and making good decisions 0 1 2 3 4 16. Concentrating, paying attention, and completing tasks 0 1 2 3 4 17. Earning money and learning how to use money wisely 0 1 2 3 4 18. Doing things without supervision or restrictions 0 1 2 3 4 19. Accepting responsibility for actions 0 1 2 3 4 20. Ability to express feelings 0 1 2 3 4

Produces a score of 0 - 80.

Resiliency Subscale


Ohio Scales Guidelines from the Texas Study

2003 study by the Texas Department of Mental Health and Mental Retardation used comparisons with CBCL and SDQ to validate the scales and to establish classification criteria for the 20-item Ohio Problems and Functioning scales:

Problems Scale

Score of 25 and above = critical impairment

Score of 17 – 24 = borderline impairment

Decrease of 11 or more points = clinically significant improvement

Functioning Scale

Score of 44 and below = critical impairment

Score of 45 – 52 = borderline impairment

Increase of 8 or more points = clinically significant improvement


Significant Improvement in Ohio Scales Problems and Functioning Scores

Benchmark Value = 70-80% of 'Impaired' Youths Should Show Significant Improvement at 6 Months

(-11 points for Problems, +8 for Functioning)

Site

# Youths with Baseline and

6-Month Assessments

# Youths with Impaired

Baseline Scale Scores

% Youth with Baseline

Impairment

# Showing Significant

Improvement

% of Impaired Showing

Significant Improvement

County 1 13 9 69% 7 78% County 2 47 36 77% 26 72% County 3 38 30 79% 20 67% County 4 11 9 82% 6 67% County 5 12 9 75% 6 67% County 6 16 14 88% 9 64% County 7 25 21 84% 13 62% County 8 73 62 85% 35 56% County 9 10 9 90% 5 56% County 10 32 28 88% 15 54% County 11 11 10 91% 5 50% County 12 16 12 75% 6 50%


Not Impaired, 154, 22%

Not Improved, 179, 32%

Improved, 381, 68%

Impaired, 554, 79%

FY 2012 Ohio Scales Clinically Significant Improvement

by Impairment at Baseline

6 Months Baseline


OSOC Youth Data 2007 – 2011 Referrals to Wraparound from Primary Sources

Ohio Scales Problems – Baseline

56% 51%

73% 71%

68% 71%

18% 14% 16% 17% 18%

12%

26%

34%

11% 12% 14%

17%

0%

20%

40%

60%

80%

Source of Referral (# referrals)

Impaired Borderline Normal

Children and youth referred by Child Welfare and Juvenile Justice are more likely to score in the normal range than those referred by other sources.


47% 49%

61% 62% 61%

50%

24%

19% 23% 23% 22%

28% 30% 33%

17% 16% 17%

22%

0%

20%

40%

60%

80%

Source of Referral (# referrals)

Impaired Borderline Normal

OSOC Youth Data 2007 – 2011 Referrals to Wraparound from Primary Sources

Ohio Scales Functioning - Baseline

Children and youth referred by Child Welfare and Juvenile Justice are more likely to score not impaired than those referred by other sources.


OSOC Youth Data 2007 – 2011 Total Enrollments by Impairment at Baseline

Normal, 408, 11%

Borderline, 516, 15%

Impaired - Function Only,

284, 8%

Impaired - Problems Only,

618, 17%

Impaired - Both Function and

Problems, 1761, 49%

Only 11% of enrolled wraparound youths score in the normal range on both Problems and Functioning scales.

Just under half of enrolled youths are critically impaired on both scales.

These data provide evidence that our referral networks are functioning well and that OSOC is providing services to appropriate families.


OSOC Youth Data 2007 – 2011 6-Month Improvement in Problems / Functioning

by Gender, Caregiver Form

59%

53% 56%

52%

0%

10%

20%

30%

40%

50%

60%

70%

Female (P=407, F=333) Male (P=943, F=800)

% o

f Im

pair

ed Y

outh

s w

/Sig

nifi

cant

Impr

ovem

ent

Gender (P = Problems count; F = Functioning count)

Improved, Problems Improved, Functioning

Females who are rated impaired at baseline show clinically significant improvement at a higher rate at six months than males on both scales.


37% 39%

37%

63% 61%

63%

0%

10%

20%

30%

40%

50%

60%

70%

6 - 11 12 - 14 15 - 18

Age Group

No improvement Improved

OSOC Youth Data 2007 – 2011 Significant Improvement, Baseline to 6 Months

by Age – Caregiver Form

229

394 279

179

194

112


Problems Scale Caregiver vs. Youth Responses, by Item (N = 1983)

2.9

1.3

2.4

2.3

2.6

1.6

0.5

1.2

0.6

2.2

2.2

0.6 0.

7

1.4

1.6

1.6

1.5

2.0

0.8 1.

0

2.5

1.2

2.1

1.9

1.9

1.1

0.5

0.9

0.6

1.8

2.3

0.5 0.

6

1.2

1.1 1.

3

1.7

1.7

1.0

1.0

0.0

1.0

2.0

3.0

Resp

onse

Val

ue (

0-5

)

Problems Scale Item

Caregiver YouthOnly five items were rated more or equally problematic by youths as by their caregivers.

Scale values: 0 = Not at All 1 = Once or Twice 2 = Several Times 3 = Often 4 = Most of the Time 5 = All of the Time


Functioning Scale Caregiver vs. Youth Responses, by Item (N = 1766)

2.5

1.9

2.3

2.4 2.

6

2.5

1.7 1.8

2.2 2.

3

1.7

2.0 2.

1

2.0

1.6

1.7 1.

8

1.9

1.5

1.8

3.2

2.5

2.7

2.7

3.1

3.1

2.3

2.6 2.

7

2.8

2.4 2.5

2.8

2.8

2.5

2.3 2.4 2.

7

2.6

2.5

0

1

2

3

4

Resp

onse

Val

ue (

0-4)

Functioning Scale Item

Caregiver YouthScale values:

0 = Extreme Troubles 1 = Quite a few Troubles 2 = Some Troubles 3 = OK 4 = Doing Very Well


OSOC Youth Data 2007 – 2011 High Externalizing/High Internalizing and Improved

95% 88% 88%

84%

70% 72% 70%

45%

26% 29% 28%

37%

44% 45% 61%

65% 68% 61%

65%

68% 58%

67% 72% 73%

86%

59%

74%

71%

0%

20%

40%

60%

80%

100%

High Externalizing High Internalizing Improved Externalizing Improved Internalizing


13%

39%

20%

87%

61%

80%

0%

20%

40%

60%

80%

100%

6 to 11 (83) 12 to 14 (69) 15 to 18 (50)

Significant Improvement

OSOC Youth Data 2007 – 2011 Impaired Youths with More than 6 School Days Missed at Baseline

26%

41% 38%

74%

59% 63%

0%

20%

40%

60%

80%

100%

6 to 11 (50) 12 to 14 (49) 15 to 18 (24)

Age Group (# Youths)

No Clinically Significant Improvement Baseline to 6 Months

Over 6 School Days Missed Under 7 School Days Missed


OSOC WFI 2008 Pilot Study Ohio Scales Change by Caregiver WFI Score

92.4 90.1 88.7 88.3 83.8

79.5

73.1

50.7 11

10

13

8

2

8

3

6 7

15

13

8

6

4

5 4

0

4

8

12

16

20

0

20

40

60

80

100

Poin

t Ch

ange

in S

cale

(A

bs V

alue

)

WFI

Sco

re

Caregiver WFI Avg. Problem Change Avg. Functioning Change

o In 2008-09, OSOC piloted the WFI 4.0 with 4 sites and 8 wraparound facilitators. The results were compared to the improvements in the Ohio Scales.

o Generally, as WFI ratings fell, so too did Ohio Scales improvement.


Future Plans for the Ohio Scales

Add borderline cases to the outcomes and appropriateness analyses.

Explore the use of the Ohio Scales as a pre-referral team screener, to provide additional information for the team that staffs each referral.

In response to research, recommend that the age range for the Youth Form of the Ohio Scales be extended down to 9-year-olds.

Add usability features to the on-line data system to improve the ability of families and staff to use and interpret Ohio Scales results.


Resources Ash, S., & Weis, R. (2009). Recovery Among Youths Referred to Outpatient Psychotherapy: Reliable Change, Clinical

Significance, and Predictors of Outcome. Child and Adolescent Social Work Journal 26, 399-413.

Davis, T., Gavazzi, S., Scheer, S., Kwon, I., Lammers, A., Fristad, M., & Uppal, R. (2011): Measuring Family Caregiver Perceptions of Support in Caring for Children and Youth With Mental Health Concerns, Journal of Social Service Research, 37:5, 500-515.

Ogles, B. M., Carlson, B. Hatfield, D. R., & Karpenkov, V. V. (2008). Models of case mix adjustment for Ohio mental health consumer outcomes among children and adolescents. Administration and Policy in Mental Health and Mental Health Services Research, 35, 295-304.

Dowell, K. A. & Ogles, B. M. (2008). The Ohio Scales Youth Form: Expansion and Validation of a Self-Report Outcome Measure for Young Children. Journal of Child and Family Studies, 17, 291-305.

Turchik, J., Karpenkov, V.V., & Ogles, B. M. (2007). Further Evidence of the Utility and Validity of a Measure of Outcome for Children and Adolescents. Journal of Emotional and Behavioral Disorders, 15, 119-128.

Carlston, D. L. & Ogles, B. M. (2006). The impact of items and anchors on parent-child reports of problem behavior. The Child & Adolescent Social Work Journal, 27, 24-37.

Ogles, B. M., Carlston, D. L., Hatfield, D.R., Melendez, G., Dowell, K., and Fields, S. A. (2006). The role of fidelity and feedback in the wraparound approach. Journal of Child and Family Studies, 15, 114-128.

Ogles, B. M., Dowell, K., Hatfield, D, Melendez, G., & Carlston, D. (2004). The Ohio Scales. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (3rd ed., Vol. 2) (pp. 275-304). Hillsdale, New Jersey: Lawrence Earlbaum.

Ogles, B. M., Melendez, G., Davis, D. C., & Lunnen, K. M. (2001). The Ohio Scales: Practical Outcome Assessment. Journal of Child and Family Studies, 10, 199-212.

Texas Department of Mental Health Mental Retardation. (2003). Validation and norms for the Ohio Scales among children served by the Texas Department of Mental Health and Mental Retardation. http://www.dshs.state.tx.us/mhprograms/RDMCAtrag.shtm


E-TEAM

The Educational Training, Evaluation, Assessment, and Measurement (E-TEAM) Department of the University of Oklahoma is a full service department with senior researchers, data analysts, technical writers, and data base developers and managers representing decades of experience in all phases of research data processing. E-TEAM also has several years’ experience as evaluation manager for the State of Oklahoma Systems of Care (OSOC) grant community, implementing its national and local evaluations as well as special OSOC collaborative studies with child welfare and juvenile justice. Belinda Biscoe, Ph.D., Assistant Vice President, Public and Community Services, at the University of Oklahoma, College of Continuing Education, is the Director of E-TEAM. E-TEAM has staff with extensive experience in tribal projects in conjunction with the American Indian Institute (also located at the University of Oklahoma). E-TEAM also has bi-lingual staff to facilitate communication with Spanish-speaking customers and subjects.

The Oklahoma Systems of Care E-TEAM evaluation staff:

Melinda Anthony, Family Interviewer

José Diaz, Data Analyst

Kelly Phillips, Research Assistant

Sharon Strait, Research Assistant

Geneva Strech, Senior Research Associate

Stephen Strech, Youth Interviewer

John Vetter, Senior Research Associate

Lisa White, Research Associate

Using the Ohio Scales for Assessment and Outcome...

Documents

Transcript of Using the Ohio Scales for Assessment and Outcome...