The Community Technical Assistance Center of New York
Bill P. Martone President WPM Consulting
The Community Technical Assistance Center of New York
To obtain Continuing Education Unit credits, please 1. Please log into the webinar system via your computer, and view the
entire presentation 2. Go to https://sswforms.es.its.nyu.edu/ 3. Create a profile 4. Select today’s webinar 5. Pay the $15 processing fee 6. Take the Knowledge Test* *The Knowledge Test will be made available to you within the week. You will receive an email with this information within 24 hours of this webinar.
The Community Technical Assistance Center of New York
December 18th Embarking on a New Path
http://www.ctacny.com/embarking-on-a-new-path-registration.html
January 15th
Culture- Driven Leadership http://www.ctacny.com/culture-driven-leadership-
registration.html
The Community Technical Assistance Center of New York
Performance Excellence: Measuring What Matters
A Leadership Webinar Series
Session 3
February 19, 2016
The Community Technical Assistance Center of New York
• Support you in identifying the strengths of your organization specific to achieving sustained positive outcomes for youth and families.
• Support you in increasing your understanding about your organizational capacity for developing or expanding community based services
• Hopefully increase your enthusiasm and understanding regarding specific strategies you can go back and utilize to improve services in your own programs
The Community Technical Assistance Center of New York
Line of Sight Management Use of Data Outcome Measurement Benchmarking Tools Q&A
The Community Technical Assistance Center of New York
We Covered: 1) Organizational Culture 2) Performance-based Culture 3) Performance Excellence – An Agency Journey 4) Leading Cultural Adaptation Materials in Appendix
Areas of Focus
Performance Excellence Areas of Focus
Proc
ess M
anag
emen
t
Leadership & Team Development
The Nature of Teams The Team Maturity Matrix
Sponsors, Coaches, Team Leaders Communication That Works
360° Evaluations Directive Supervision
Leadership as a System Leadership Assessment & Measurement
Process Management
The Process Organization Process Measures
Suppliers, Inputs, Process, Output, Customers (SIPOC) Rapid Improvement Technology
5 Step Improvement Process Plan, Do, Check, Act (PDCA)
So Today We Will Look at Line of Sight Management Data, Outcomes and Data Collecting Tools
Line Of Sight Management
The Vital Few People Aligned with Goals & Measures Linkage to Business Plans Fact Based Decision Making
Catch Ball Useful Information vs. Data Overload Reporting and Presentation
Div/Dept Plans, Goals & Metrics
Mgmt. Plans, Goals & Metrics
Team Plans, Goals & Metrics
Company Metrics
And Results
Sr. Exec Team
Executive Teams
Management Teams
Teams
Org/Dept Metrics
And Results
Mgmt. Metrics And
Results
Team Metrics And
Results
Strategic Plan
Company Plan, Goals &
Metrics
Line of Sight Architecture
14
Are You Drowning in Data?
Data Rich – Information Poor
Focus on Data • Data Collection
• What to Measure • Operational Definitions • Data Sources • Collection Methods • Frequency • Data Display
Data-driven decision making
Meaningful Data and Measures
Satisfaction Survey
Training Pre-Post Tests
Consumer Focus
Satisfaction Surveys
Safety Data
Financial Results
IT Systems
Turnover and
Leaves
Audit & Compliance
Results
Mindful Measurement Drill down to Customer Focus
Youth and Family
Engagement
Staff Deployment of Principles
Fidelity to Core Practice
Principles
Youth and Family
Satisfaction
Focus Groups
Satisfaction Surveys
Utilization Data
Phone Calls w/ Families
Complaints & Grievances
Follow-Up Studies
MEASUREMENT DOMAINS DATA SOURCES
19
Our Cycles of Improvement
Line of Sight Reporting at
Leadership Meeting
Annual and Strategic Planning
Improvement Recommendations
Deploy Implementation
Plans
Data Monitoring, Analysis, and Consolidation
Satisfaction Survey
Training Pre-Post Tests
Consumer Focus
Satisfaction Surveys
Safety Data
Financial Results
Turnover and
Leaves
IT Systems
Audit & Compliance
Results
October/November Individual Performance
Development Plan Data is Gathered
Environmental Scan Completed by ELT
December/January Review Environmental Scan Information and
Data* with LT
February LT Members Present Data and Conduct a S.W.O.T. with Each
Agency Department Then LT Review S.W.O.T. Data and Chooses Top 18
in Each Category
March/April LT and Business
Performance Begin Workforce Planning and
Budgeting ELT Selects Organizational Goals and Strategies and
Shares with LT
May/June Budget is Reviewed with
Board of Directors Departments Select Their
Annual Goals Aligned with Organizational Goals
July Budget and
Organizational Goals Approved by Board of
Directors
September Organizational Goals and Budget Communicated
Throughout Organization
Strategic Planning Cycle
Strategic Planning
California Benchmarking Initiative
Actually Two Initiatives
California Alliance
Benchmarking for Financial S w vival And Organizational
Excellence
A Practice lmprovement Project Why Benchmarking • Benchmarking is a powerfu l management tool for moving organizations toward financial, operational and
practice optimi zation Benchm:uking Objectives • Data for Advocacy • Individualized benchmarking reports organizations can use to compare their performance to others in the
state and throughout the U.S . • Organizational improvement by identifying potential best practices What is Benchmarking • Measuring and comparing results of key work processes with those of best performers • Giving performa nce data meani ng by providing an external context for understanding the information,
identifying both strengths and opportun ities for improvemcnt, and suggesting best practices by distinguishing top performe rs from others
Benchmarking Data • Confidential, easy to complete on-line surveys already in use by the National Alliance for Strong and Familie s
and Communities as well as CCCMHA • National Alliance survey contains administrntivc, residential, foster care/treatment foster care, outpatient,
family preservation, juvenile justice, and e arly chi ldhood/education items • CCCMHA survey contains financial, organizational, and clinical i tems • New item development for practice areas not already included • Organizlltioal climate survey also available for confidential online completion by employees
Benchmllrking Reports • National norms and California norms if at least 15 Alliance agencies in a specific area participate • Easy to use, individualized reports provide detailed percentile r:mkings for each din1ension mellsured, a very
clear picture of strengths and weaknesses relative to other organizations quickly emerges • Timel y standard and executive reports
Favorabllity Dashboard Overall Clinical Benchmarks
Favorable Unfavor.ble 16.67'4 22.58 Unt.vo11ble
33.33
Neutral Nr:utrel S0.00%
58.06"
Fina ncial Benchmarks Operational Benchmarks favonble Unfavorable
9.09 Unfavorable 14.29 Favorable
27.27'4 28.57
Neutr.I 57.14%
Percent Metrics Submitted 45.50%
Program ID: 999 Page 1of 3 Ma_rch 27, 2008
Sample Report - Restraint/Seculsion
Relevant Agency Information which may
affect the outcome Agencies
identif ied w ith a confidential
ID number
Data provided over a rolling 3-year period for trending
Reported data converted to make for an easy comparison
McKinley Bench Marking Project
Sample Report – Filtering Data
McKinley Bench Marking Project
The Community Technical Assistance Center of New York
The Community Technical Assistance Center of New York
Quality Indicator Metric Definition Performance Rating Trend Ratings
Favorable Comparable
Unfavorable Q1 Q2 Q3 Q4
Reliable Outcomes Restrictiveness of Living
Environment Scale (ROLES)
Percentage of youth discharged to a home-like environment. 80-100% 60-79% Less than 60%
Consumer Satisfaction
Youth, Family, and Adult Satisfaction Survey
Consumer’s overall mean score on satisfaction with services where 1=low satisfaction and 5=high
satisfaction.
4.0-5.0
3.0-3.99
2.0-2.99 Collected annually
Practice Process Incident Report Overall total incident rate, includes all incident types compared to the
historical average rate.
Below 1 standard deviation
Within 1 standard deviation
Above 1 standard deviation
Fiscal
Annual Forecast
Agency overall net operating income based on preliminary data.
Break even or surplus
Not Applicable
Deficit
Hathaway-Sycamores’ 4-Square of Overall Quality Indicators FY 2013-2014 – Quarter 2 (Oct - Dec)
Reliable Outcomes Consumer Satisfaction
Fiscal Practice Process
Report prepared by the Evaluation and Research Department
The following indicate the overall favorability trends for items:
A Data Dictionary is an Absolute Must!
The Community Technical Assistance Center of New York
The Community Technical Assistance Center of New York
Possible Residential Treatment Center Measures
• Examples
• Recidivism • Success in Community • Seclusion & Restraints • Satisfaction – Youth/Families • CANS/CAFAS/ROLES • Family Finding/Natural Supports • Follow-Up Studies
The Community Technical Assistance Center of New York
Residentially Based Services
Outcome Measures
Outcome Measures
Data Sources/ Data Collection
Frequency of data collection
Data Providers
Outcomes for Children and Families
1. Achievement of permanency: Children at RBS exit with legal permanency (adoption, guardianship and reunification), and any type of placement episode termination
CWS/CMS
Annual
BIS
2. Average lengths of stay (in group care and entire RBS period)
CWS/CMS Annual BIS
3. Rates of re-entry into group care and foster care of children enrolled in the RBS program
CWS/CMS Annual BIS
4. Analyses of the involvement of children or youth and their families in services planning and treatment (Do children and families have a sense of “voice and choice” in their treatment experience?
CANS, YSS, YSS-F
Semi-annual
RUM/Provider
5. Client satisfaction YSS, YSS-F Semi-annual Provider
6. Child safety: Substantiated maltreatment while at home or in group care during RBS period
CWS/CMS Annual
7. Child well-being: Positive placement changes and # of placement moves
CWS/CMS Annual
8. Child educational progress Child’s case file Annual Provider
9. Child and family voice and choice Child’s case file Annual Provider
10. The existence of a connection with a caring adult Child’s case file Annual Provider
Outcome Measures
Data Sources/ Data Collection
Frequency of data collection
Data Providers
Systems Operation
11. Use of the program by the County On-going process County
12. The operation of the program by the private nonprofit
On-going process Provider
Fiscal Outcomes
13. Payments made to the private nonprofit agency by the County
Fiscal Workgroup
14. Actual costs incurred by the nonprofit agency for the operation of the program
Fiscal Workgroup
15. The impact of the program on State and County AFDC-FC program costs
Fiscal Workgroup
16. The impact of the program on State and County Early Periodic Screening, Diagnosis, and Treatment (EPSDT) Program costs
DMH
17. The impact of the program on State and County Mental Health Services Act (MHSA-Proposition 63)
DMH
Outcome Measures
Data Sources/ Data Collection
Frequency of data collection
Data Providers
Safety S1. 99.6% of the children/youth who are enrolled in RBS do not have any new substantiated allegations as specified in California Health & Safety Code, Section 1522(b) while receiving services under this contract
CWS/CMS
Quarterly
BIS
S2. 100% of Corrective Action Plans (CAPs) are submitted on time and successfully implemented, including facility and safety deficiencies
Corrective Action Plans/Auditor Controller
Reports
Annual
Provider
S3. 98% of children/youth are free from child-to-child injuries while in the residential site.
Child’s Case File/Facility Review Reports/
SIR/I-Track
Annual
Provider & Wrap Admin
S4. 94% of the children/youth who are enrolled in RBS do not have any new substantiated allegations within one (1) year after graduating from RBS
CWS/CMS
Annual
BIS
Permanency P1. 75% of youth that graduate from RBS will not have a subsequent out-of-home placement after six (6) months
CWS/CMS Semi-annual BIS
P2. 85% of families whose children/youth graduating from the RBS Demonstration Project continue using community based services and supports six (6) months after graduation
Follow-up Reports/
POC
Bi-annually
(Dec & June)
Provider
P3. CONTRACTOR will maintain an overall average length of stay of ten (10) months or less (in Residential)
CWS/CMS Child’s Case File
BIS Provider
P4. 80% of children/youth enrolled will have at least five (5) adult family members and fictive kin (non-relative) identified within ten (10) months of enrollment
POC Child’s Case File
Provider
P5. CONTRACTOR will facilitate 100% contact of approved connections
POC Child’s Case File
Provider
Outcome Measures
Data Sources/ Data Collection
Frequency of data collection
Data Providers
Well Being
WB1. 70% rating of family and youth satisfaction with services
YSS/YSS-F WFI-4/CAFAS
Semi-annual
Provider
WB2. 70% of youth demonstrate improvement on the behavioral/well-being measures
CAFAS
Semi-annual
Provider
WB3. At least 51% of CFT is comprised of informal supports
POC Child’s Case File
Semi-annual
Provider
WB4. 75% of children/youth maintain at least an 80% school attendance rate or improved attendance rate from previous quarter.
Child’s Case File
School Report Card
Quarterly
Provider
Displaying Data
The Community Technical Assistance Center of New York
Tools Built and/or Utilized
Meeting Minutes Format
Fish Bones
Charters
Five Step Model
Storyboard
Holding the Gains Four
Panel
Root Cause Prioritization
Matrix
Project Life Cycle
Meeting Agenda Format
Project Out Briefing
Templates Kaizen
Action Item List
Improvement Plan Format
Meeting Evaluation Form & Graphing Tool
Project Evaluation Worksheet
SIPOC
The Community Technical Assistance Center of New York
The Community Technical Assistance Center of New York
The Community Technical Assistance Center of New York
The Community Technical Assistance Center of New York
Kaizen (改善)
Kaizen is the practice of continuous improvement. Kaizen was originally introduced to the West by Masaaki Imai in his book Kaizen: The Key to Japan’s Competitive Success in 1986. Today Kaizen is recognized worldwide as an important pillar of an organization’s long-term competitive strategy. Kaizen is continuous improvement that is based on certain guiding principles:
Good processes bring good results Go see for yourself to grasp the current situation Speak with data, manage by facts Take action to contain and correct root causes of problems Work as a team Kaizen is everybody’s business And much more!
One of the most notable features of kaizen is that big results come from many small changes accumulated over time. However this has been misunderstood to mean that kaizen equals small changes. In fact, kaizen means everyone involved in making improvements. While the majority of changes may be small, the greatest impact may be kaizens that are led by senior management as transformational projects, or by cross-functional teams as kaizen events.
Kaizen Rapid Business Improvement Process (1- 5 weeks)
Data Collection
Identify Waste And Defects
(Opportunities For Improvement)
Analysis of Causes
(Fishbone)
Root Cause Analysis
Leverage Points
Preliminary Solutions
Improvement Plan Alternative Solutions
1.
2.
3.
4.
Solution
Savings $$$
Cost
Plan
Why?
Why?
Why?
Why?
Why?
OFIs OFI
Tools People
Layout Structure
Root Cause
OFIs
Cause Cause
Cause Cause
Cause
Cause
5 Whys
Final Solutions
Story Board
57 57
We Utilized The Team Handbook To Learn About Tools The Following Examples Are From The Handbook
Ishikawa Cause & Effective Diagram - Fishbone
The Tool • Developed by Kaoru Ishikawa • Diagram called fishbone because it looks like a fish skeleton with head spine and bones • Unlike others this tool does not analyze data
• Helps people organize their ideas and theories about causes • These theories later become verified with data
The Fishbone • The problem under investigation is described in a box at the head of the diagram • A long arrow pointing to the head is the spine and indicates the problems feed into
the spine that might cause the problem at the head • The larger bones feeding the spine represent the main potential causes of the problem • The smaller bones represent deeper causes of the larger bones • Greatest challenge is to have the bones show cause and effect relationships
Ishikawa Cause & Effective Diagram - Fishbone
Ishikawa Cause & Effective Diagram - Fishbone
An Easy Problem – Bad Tasting Pizza
Ishikawa Cause & Effective Diagram - Fishbone
Problem – Appointment No Show
Ishikawa Cause & Effective Diagram - Fishbone
Tools Most Likely Used In The 7 Step Method Discussed Last Time
Had to focus the organization around the importance of using data and figure out what was
important for us to know – vital few – data dictionary Data Collection & Outcomes Reporting Takes Resources
• Dedicated Research & Evaluation Staff • Funded primarily through our Mental Health Contract • Utilized a number of graduate students in the department with small stipends
Key focus coming out of 2008 was assuring adequate resources for the agency
• Paying attention to staff’s billing became labeled as “Its all about the money”
As reports were produced and shared transparency throughout the agency occurred • Everyone knew how everyone else was doing
While the tools were extremely useful it took time to train staff in their use and help staff
understanding the right tools to use for a given analysis
The Kaizen process took a significant investment of staff time up front but cut significant amounts of time off the duration our our projects
Obstacles & Challenges To Using Data/Tools/Outcomes/Line of Sight Reporting
’s
Contact Information
William P. Martone President WPM Consulting, Inc. 1516 Beech Street South Pasadena, CA 91030 Cell Phone: 626-831-6850 E-Mail: [email protected]
The Community Technical Assistance Center of New York
Yvette Kelly [email protected]
Appendix
Some Excerpts From Hathaway-Sycamores Website Comprehensive Outcomes Report
www.hathaway-sycamores.org
Top Related