1

Heartland CIT: Toward Comprehensive Evaluation

International Crisis Intervention Team Conference 2010

San Antonio, TexasJune 1, 2010

Shinobu Watanabe-GallowaySteve Spelic

2

Acknowledgement

• Consumers, families, and advocates who made the implementation of Heartland CIT program possible

• Officers and agency leaders who have participated and embraced the program

• Providers and other professionals who have helped the planning and implementation

• CIT Council members and all other individuals and organizations involved

• University of Nebraska Medical Center College of Public Health Evaluation Team

3

Workshop Objectives

1. Learn about a comprehensive evaluation approach used by Heartland CIT Program

2. Learn how Heartland CIT used evaluation data to improve curriculum and identify training needs

4

Heartland CIT Program

U.S. Census 2009 Estimates-Nebraska: 1.8 million -Douglas County: 510,199 (Omaha: 419,949)-Sarpy County: 153,504

6

Background

• Nebraska Chapter of NAMI coordinated an effort for diversion programs including CIT– Consumers/families have played an important role

in planning and implementation• CIT Council established in December 2005• Began with 2 workshop per year with three

agencies to 3 workshops per year

7

8

CIT Participating Agencies

Police De-partments (208) 70%

Douglas Co. Sheriff's Dpt

(38) 13%

Douglas Co. Corrections

(30) 10%

Probation (9) 3% Other (12) 4%

Police Dpt’sOmaha 160Papillion 14Bellevue 12Boys Town 6Lincoln 4Norfolk 4LaVista 2Council Bluff 2Plattsmouth 2Blair 1Fremont 1Total 208

“Other”Metro Comm College 3NE Law Enf Training Ctr 1911 Center 6Emergency Coordinators 2Total 12

9

Heartland CIT Program EvaluationStep 1:

Identify purpose and objectives

Step 2:Choose evaluation

methods

Step 3:Collect data

Step 4:Review results

Step 5:Use results to improve

program

10

Wksp 1

Wksp 2

2006 2007 2008 2010

Wksp 3

Wksp 4

Wksp 5

Wksp 6

Wksp 7

Wksp 8

Wksp 9

Wksp 10

Wksp 11

Wksp 12

2009

Onsite Paper & Pencil Survey

Follow-up Mailed Survey

Follow-up Internet Survey

2007 January 2008 October

2009 Jun - Dec

Focus Groups

Encounter Forms

2010 Apr - May

11

Wksp 1

Wksp 2

2006 2007 2008 2010

Wksp 3

Wksp 4

Wksp 5

Wksp 6

Wksp 7

Wksp 8

Wksp 9

Wksp 10

Wksp 11

Wksp 12

2009

Onsite Paper & Pencil Survey

12

On-Site Survey

• To evaluate participant’s assessment of – Usefulness of each session in terms of content covered and

instruction methods used– Instructor / instruction – Logistics (e.g., facility, communication prior to workshop)– Benefits of overall CIT workshop

• Data has been used to:– Improve the overall curriculum (changing schedule, time

allocation, new topics added, etc)– Refine materials for specific sessions– Identify alternative instructors in a few cases

13

Onsite Questionnaires

- Paper-and-pencil survey- Curriculum / instruction survey

Session Survey1-page survey completed after each session

Summary Survey2-page survey completed at the end of 1-week workshop.

Session Survey Example

14Summary Survey Example

15

Onsite Surveys: Major Findings

• About 70% of officers reported that they did not have adequate knowledge prior to the workshop– Overview of mental illness, substance abuse– Older adults / children, developmental disabilities– Active listening, crisis de-escalation, suicide

prevention– Community resources– Cultural aspects

16

Major Findings (cont’d)

• More than 90% of officers reported enhancement of knowledge from all 21 topics except 3 (children – 89%; substance abuse 86%; co-occurring 78%)

• Sessions that received very positive ratings were:– Consumer / Family Perspectives– Scenario-based training – Hearing Voices– Site Visits

• Workshops provided opportunities for interaction and gaining insight about mental illness

Addressing CIT ClassThe Chief of Police and the Douglas County Sheriff addressing the class at the beginning of training.

Family PerspectivesFamily members sharing their stories with the class.

18

Hearing Voices & Virtual Hallucination SessionsTwo of the most popular sessions of CIT Workshop.

19

Scenario Based Training:Two officers just beginning a scenario and coming to a person’s apartment. The lady in the picture is one of our actors who is portraying a client in crisis. Officers are just showing up “on the scene” to deal with a mental health crisis.

Interaction – Outside of Scenario Based Training Shows interaction between actors of the scenario training, an officer, and a consumer (the officer and consumer are the evaluators of the scenarios) - this is occurring just outside the rooms where we do the scenario training.

20

Opportunities for Interactions:During the workshop, especially around scenario-based sessions, lots of interactions and conversations happened among officers, between officers and consumers and workshop coordinator.

21

Wksp 1

Wksp 2

2006 2007 2008 2010

Wksp 3

Wksp 4

Wksp 5

Wksp 6

Wksp 7

Wksp 8

Wksp 9

Wksp 10

Wksp 11

Wksp 12

2009

Onsite Paper & Pencil Survey

Follow-up Mailed Survey

Follow-up Internet Survey

2007 Fall/Winter

2008 Fall

22

5-Month Mailed Follow-up Survey of Sep 2006 Graduates (N=20 / 23) (January 2007)

• I used different questions and handled the person differently than in the past.

• I have a more calm and open approach with the person in crisis.

• I gave the party involved more time to process the questions that I asked.

• I have asked about their book if they have one or if they have a plan as to what we can do to assist them better especially when they have a history of documented mental illness.

Figure 1: Calls Received from Individuals with Mental Illness

0

1

2

3

4

5

6

7

8

9

1 to 4 5 to 9 10 to 14 15 to 19 20 or more

Number of Calls Received

Num

ber o

f Offi

cers

23

Figure 3: Percentage of participants that would like additional training in these topics

0 10 20 30 40 50 60 70 80 90

Overview of Mental illness

Medications and Side Effects

Age and Medical Conditions

Intervention with Children and Adolescents

Wellness Recovery Action Plan

Active Listening & Crisis De-escalation

Crisis Intervention, Negotiation and Tactical Aspects

Consumer Perspectives

Preventing Suicide

Site visit

Co-occurring Disorders

Developmental Disabilities

Assessing Treatment Resources

Managing Family, Friends, and Bystanders

Family Perspective

Effects of Culture on Psychiatric Crisis

Mental Health Law & Jail Diversion

CIT Officer Panel & Integration CIT Techniques

Hearing Voices

Scenario Based Training

Session Title

Percent

24

25

Internet Follow-up Survey of OPD Graduates (N=24/104) (October 2008)

• Survey monkey makes it easy to collect and analyze data

• The downside is low response rate

• Reminder and notice from the department may increase response rate

0 1 2 to 5 6 to 10 11+0

2

4

6

8

10

12

Figure 2: Calls Received from Individuals with Mental Illness

Number of Calls Received

Nu

mb

er

of

Off

ice

rs

26

Usefulness of CIT Classes

Strongly Disagree Disagree Neither Agree

Strongly Agree

I am better able to assess whether the person has a mental illness or not.

0.0% 4.3% 4.3% 56.5% 34.8%

I am more patient with the person who has a mental illness.

0.0% 0.0% 4.3% 43.5% 52.2%

When dealing with a person who has a mental illness, I have a better understanding of their needs.

0.0% 0.0% 4.3% 52.2% 43.5%

I am better able to communicate with a person who has a mental illness.

0.0% 0.0% 13.0% 43.5% 43.5%

I am better in my use of the de-escalation skills.

0.0% 0.0% 13.0% 52.2% 34.8%

I am better able to use the active listening skills than before.

0.0% 4.3% 13.0% 39.1% 43.5%

27

Wksp 1

Wksp 2

2006 2007 2008 2010

Wksp 3

Wksp 4

Wksp 5

Wksp 6

Wksp 7

Wksp 8

Wksp 9

Wksp 10

Wksp 11

Wksp 12

2009

Onsite Paper & Pencil Survey

Follow-up Mailed Survey

Follow-up Internet Survey

2007 January 2008 October

2009 Jun - Dec

Encounter Forms

28

Incidence Tracking Form

Pilot tested for Douglas Co. Sheriff’s Dept and Omaha PDJune – Dec. 2009

135 forms completed

CIT Non-CIT

Total

Douglas Sheriff

37 71 108

OPD 25 2 27

Total 62 73 135

Consumer Demographics

≤10 11 to 20 21 to 30 31 to 60 61 to 70 71 to 800

10

20

30

40

50

60

70

80

Age (yr)

Num

ber o

f For

ms

Female51%

Male49%

Chart Title

White87%

African A

meri-can

10%

Hispanic4%

30

N = 135 %Known person?Yes 44 32.6No 55 40.7Don't know 33 24.4Repeat call ?Yes 6 4.4No 93 68.9Don't know 29 21.5

N = 135 %Parent 31 23.0Self 21 15.6Partner/spouse 16 11.8Other family member 14 10.4Hospital/medical staff 14 10.4Friend/Acquaintance 10 7.4Boyfriend/girlfriend 8 5.9Otherwise unknown 6 4.4Sibling 6 4.4Police Observation 5 3.7Business Owner 2 1.5Don't know 2 1.5

Reporting Party

Known Person?

Repeated Call?

N = 135 %Suicide threat or attempt 73 54.1Disorderly/disruptive 28 20.7Neglect of self-care 31 23.0Drug-related offense 10 7.4Public intoxication 6 4.4Nuisance 6 4.4Theft 2 1.5

N = 135 %Threats/violence/weapon? Yes 20 14.81No 112 82.96Don't know 2 1.48Incident injuries ?Yes 20 14.81No 115 85.19Drug/alcohol involvement? Yes 54 40 No 72 53.3Don't know 9 6.7Med compliance reported ? Yes 29 21.5 No 106 78.5

Nature of Incident

(“Check all that apply”)

Diagnosis

(“If known” / “Check all that apply”)

N = 135 %Depressed 51 37.8Disoriented/confusion 32 23.7Disorganized speech 28 20.7Belligerent or uncooperative 27 20.0Manic 17 12.6Delusions 14 10.4Unusually scared or frightened 10 7.4Hallucinations 4 3.0

Behaviors at time of Incident

(“Check all that apply”)

N = 135 %Disoriented/confusion 32 23.7Delusions 14 10.4Hallucinations 4 3.0Disorganized speech 28 20.7Manic 17 12.6Depressed 51 37.8Unusually scared or frightened 10 7.4Belligerent or uncooperative 27 20.0

Encounter Outcomes

Douglas Sheriff OPDN = 108 % N = 27 %

Transported to treatment facility 92 85.2 15 55.6

No action/resolved on scene 8 7.4 3 11.1

On-scene crisis intervention 5 4.6 6 22.2

Arrested 4 3.7 3 11.1

Police notified CM or MHC 3 2.8 1 3.7Outpatient/Case management referral 1 0.9 0 0.0

34

Consumer Characteristics: Sheriff’s Dpt vs. OPD (CIT officers)

• Similarities: Age, gender, race, diagnosis• Differences in disposition of consumers

– Disorderly /disruptive (Sheriff: 41% vs. OPD:13%)– Neglect of self care (Sheriff: 30% vs. OPD: 14%)

35

CIT vs. Non-CIT (Sheriff’s Dpt Data)

• Disorderly/Disruptive disposition – Sheriff: 40.5% vs. OPD: 12.7%

• Female consumers – Sheriff: 64.9% vs. OPD: 45.1%

36

Wksp 1

Wksp 2

2006 2007 2008 2010

Wksp 3

Wksp 4

Wksp 5

Wksp 6

Wksp 7

Wksp 8

Wksp 9

Wksp 10

Wksp 11

Wksp 12

2009

Onsite Paper & Pencil Survey

Follow-up Mailed Survey

Follow-up Internet Survey

2007 January 2008 October

2009 Jun - Dec

Focus Groups

Encounter Forms

2010 Apr - May

37

Focus Groups – Preliminary Results

• Three focus groups– OPD– Douglas County Corrections– Douglas County Sheriff’s Department

• Conducted in March – April 2010• 4 – 10 people participated for each group• Five Questions

38

1. Skills or knowledge areas gained

1. Awareness and knowledge of mental illness

2. Knowledge of medication3. Communication and de-

escalation skills 4. Attitudes (tolerance and

openness) 5. More options for handling

crisis situations 6. WRAP plan helped to

establish trust7. Learned about community

resources and organizations

2. Training needs

1. Continuing education on new medications/ updates on mental illness and treatment

2. Expansion on communication techniques

3. Training tailored to meet specific needs of agencies

4. Intervention with youths 5. Section that addresses

mental health of officers (to remove stigma)

39

3. Refresher Course?

1. Follow up within 6 months2. Continuing education

annually 3. Give participants a chance to

go back to CIT workshop and help train others

4. Networking opportunities5. Update contact information

for current resources6. Video-tape session to make

the CIT available to officers who cannot attend

4. Youth Specific CIT Program Needed?

Yes because it will 1. Remove a current gap in the

CIT program2. Prepare officers for

interacting with youth3. Help remove stigma and

labeling of youth, and improve acceptance of youth with mental illnesses

4. Could be a 1-day program that would attract more participants

40

5. Other Suggestions

1. Have a ‘Corrections’ specific class: different needs and many facilities could participate

2. Develop a better policy for dealing with hospitals: need more standardization between hospitals and officers need to know which one is best equipped to handle different situations.

3. More information on interacting with the elderly.

4. More applied, hands-on learning.

41

Future Directions

• Information from multiple sources – Key stakeholders including consumers, families, providers

• System-level analysis– Economic analysis – Impact on community

42

Community

Criminal Justice

Law Enforcement

Providers

Consumers and Families

Community