Lessons learned from the Evaluation of Baltimore Expanded School Mental Health: Challenges and...

Lessons learned from the Evaluation of Baltimore Expanded School Mental Health:

Challenges and Solutions

Georgetown University National Technical Assistance Center Webinar SeriesOctober 21, 2010

Making the Case for Evaluation of School Mental Health Programming: Strengthening Partnerships for Positive Outcomes

Bruno J. Anthony, Ph.D.

Georgetown University Center for Child and Human Development

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY

Collaborators• Baltimore City Public Schools

– Louise Fink– Ike Dibor – Shawn Hales

• Baltimore Mental Health Systems– Jane Paplinger– David Jones– Denise Wheatley-Rowe

• Georgetown Center for Child and Human Development– Bruno Anthony– Joyce Sebian– Courtney Holland– Clare Dunne


Overview BESMH• Baltimore ESMH a recognized state and national leader for more than 20

years.

• ESMH services have complemented and supplemented the services provided by Baltimore City Public Schools (social workers, psychologists, and guidance counselors)

• Currently, prevention and treatment services provided in 106 schools (approximately half of the schools in Baltimore City) through 12 provider agencies and 106 clinicians.

• Intended to address barriers to learning to enable students to make better use of educational programs in their schools and to provide an alternative to mental health services within structure of Special Education System

• Funding: Braided funding from school system, Department of Human Resources, Baltimore Substance Abuse System, Baltimore Mental Health System.


Evaluation of BESMH• Purpose: Assess the impact of a comprehensive ESMH program

– Test assumptions that ESMH is beneficial in terms of academic outcomes

– Inform programming

– Inform ongoing evaluation and sustainability efforts.

• Funding– Planning: Internal funding from BESMH budget

– Implementation: State of Maryland Mental Health Transformation State Incentive Grant (TSIG)

Demand for academic accountability heightens the need for studies examining the specific impact of ESMH


Challenges and Approaches: Developing a Meaningful and Feasible Plan

Key Informant Input– Focus Groups (administrators, program directors,

clinicians/providers, caregivers, youth)– Consultation (advisory committee, monthly program directors

meetings, leaders in the field)– School system partners (DREAA, Student Support Services, leaders)

Goals– Identifying important barriers and challenges – Identifying the appropriate and relevant outcomes – Incorporating content expertise in understanding ESMH in

Baltimore– How to use the evaluation findings to improve the ESMH program

and for advocacy.

A participatory approach


Challenges and Approaches: Examining Outcomes at Individual Level

. Employed the “zipper” for linking separate confidential data sets that

contain personal identifying information while preserving anonymity.– Safe analyst to link and process data who is not part of either the

research team, the agency with overall administrative oversight over the ESMH program or the school system.

– Providers send service use data set with ID numbers sent to the “safe analyst,” who has no access to student names.

– School system sends demographic and academic/school functioning with ID numbers to safe analyst.

– Safe analyst uses ID numbers to merge the two data sets then encrypts ID numbers

Risk of connecting private school and ESMH data


Challenges and Approaches: Capturing Spectrum of ESMH Services

Electronic data collection– Developed in a participatory process

• Individual: For each date of contact, the type(s) of services and, at the end of each month, the focus of treatment and the type of the interventions.

• Group: attendance in prevention and treatment groups.• Universal: four types of prevention activities.

– Training: direct contact, manual and web-based

Service logs were not capturing the range of promotion, prevention and treatment work carried out by clinicians.


Extent and Type of Services

ESMH Clinicians touch an enormous number of students, caregivers and school staff, extending the range of clinical service that could not happen in more traditional settings.Clinicians had direct contact with more 6254 students -1/5th

of students attending the 106 schools that support ESMHA large part of the clinicians work involves indirect,

collateral services


Percentage of Students by Grade


Percentage of Service Types


Percent of Direct and Indirect Services


Service Focus

• Clinicians focus on issues that disrupt a student’s engagement with school

• Clinicians for the most part report using approaches with an evidence base

• Family/caregiver engagement is challenging


Percentage of Focus Areas Coded


Percentage of Intervention Types Coded


Challenges and Approaches: Comparison Groups

Quasi-experimental strategies to support causal influences of ESMH.

Approaches:– Contact Groups

– Matched Comparison Groups

– Dosage Groups

As is often the case with on-going, community-based programs, a full-fledged experimental (random-assignment) design was not practical, feasible or ethical.


E-E: Students receiving ESMH services in an ESMH schoolNoE-E: Students NOT receiving ESMH services in an ESMH schoolNoE-NoE: Students NOT receiving ESMH services in a school without ESMH

Distribution of Students in Different Dosage Groups


Matched Comparison Group Formation• Propensity scores

– Probability of a student in a school with an ESMH clinician receiving ESMH services, given a set of background variables for that individual.

– Determined background risk factors that predicted participation in ESMH using Binary Logistic Regression (2007-08 attendance, suspensions, MSA Reading and Math scale scores).

– Children receiving ESMH would have high propensity scores based on the set of background variables . Students with similar background characteristics but not receiving ESMH services should also have high propensity scores.

– Propensity score of each ESMH student matched with a student drawn randomly from those students not enrolled in ESMH to form Comparison Group.


Preliminary Overall Results

A greater percentage of students attending a school with ESMH services and accessing these services showed– Improvement over the course of the year in Math and

English benchmarks.

– Decreases in suspensions


Challenges and Approaches: Outcome Measurement Framework

Integrate data needs and system pressures of different stakeholders.– Use available data records, outcome measurements already in place,– Stakeholders recognized the imperative of demonstrating

effectiveness in the school setting with selected mediating/moderating contextual variables.

– The importance of looking at school outcomes (e.g., academic success, promotion, absences and suspensions) to ensure sustainability

Devising an outcome measurement framework that satisfies different stakeholders without overburdening clinicians


Evaluation Contributions to Sustaining ESMH

Building evaluation and research model – participatory process Building infrastructure for ongoing evaluation and research– Service Trends

• Impact on programming decisions• Impact on financial models• Training

– Leveraging other initiatives. Utilizing evaluation data to make the case for ESMH program– Case for Ancillary and Supportive Services

• Teacher Consultation • Care Coordination• Family contacts

– Case for School-related Benefits • Improvement in academic measures• Positive effects on school attendance and behavior


ContactingContacting UsUs

BESMH Evaluation Team:

Bruno Anthony, Ph.D.Professor and Director of Research and Evaluation,

Email: [email protected]

Joyce Sebian MS. Ed.Senior Policy Associate,

Email: [email protected]

Georgetown University Center for Child and Human Development

Department of Pediatrics3300 Whitehaven Street, NW, #3300

Washington, DC 20007

27

Lessons learned from the Evaluation of Baltimore Expanded School Mental Health: Challenges and...

Documents

Transcript of Lessons learned from the Evaluation of Baltimore Expanded School Mental Health: Challenges and...