Data: Telling a Story … A Walk Through Meaningful Data Analysis
description
Transcript of Data: Telling a Story … A Walk Through Meaningful Data Analysis
FosteringCourtImprovement.org
Data: Telling a Story …A Walk Through Meaningful Data
Analysis
Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results
Andy BarclayStatistician and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law
Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006
FosteringCourtImprovement.org2
Overview How Knowledge of the Past and Present
can Help you Shape the Future• Data, Data, Data Isn’t Yada, Yada, Yada• Court Performance Measures• AFCARS
The Easy Win vs. Sea Change Getting Real, Getting Local, Getting
Context Results that Matter to Kids & Families
FosteringCourtImprovement.org3
Shaping the Future
Why Data, Data, DataIsn’t
Yada, Yada, Yada
FosteringCourtImprovement.org4
Lessons learned or missteps repeated?
Lunchroom lore or documented results?
Shaping the Future
FosteringCourtImprovement.org5
Shaping the Future
“They always say that time changes things, but you actually have to change them yourself.”
Andy Warhol
FosteringCourtImprovement.org6
Do you know where your caseload has been?
Do you know where it is?
Do you know where it is going?
Shaping the Future
FosteringCourtImprovement.org7
Shaping the Future
Am I on the road to success or on a “frolic and detour”?
(Only the data know.)
FosteringCourtImprovement.org8
Shaping the Future
“Georgia Judge leads stampede out of homes.”
FosteringCourtImprovement.org9
Shaping the Future
“Illinois Judge ‘doesn’t have time’ to return children to parents.”
FosteringCourtImprovement.org10
Shaping the Future
Take off the blinders.Or
Load and Learn.
FosteringCourtImprovement.org11
Shaping the Future
Where to Start?
FosteringCourtImprovement.org12
Court Performance Measures
The three leading legal/ judicial organizations, with a grant from the Packard Foundation, have created and pilot-tested court performance measures that complement those used in the CFSR process so that agencies and courts can collaborate in the measurement of their performance and tracking of their own progress in improving safety, permanence, and timeliness for children coming before them.
FosteringCourtImprovement.org13
Court Performance Measures
(Building a Better Court) Reinforce current federal reform
initiatives, including CFSR and PIP’s Helps to identify areas of court reform
essential to better outcomes Assist the CIP by providing courts with
tools to engage in systematic process of court performance measurement
FosteringCourtImprovement.org14
Court Performance Measures
“Performance measures are the yardstick by which courts and child welfare agencies can measure the success they are achieving and the progress they are making.”
Building a Better Court
FosteringCourtImprovement.org15
Court Performance Measures
Safety: Are there new allegations while in foster care?
Permanency: What was the exit from foster care and how many came back?
Due Process: Is there notice and representation?
Timeliness: How long between hearings?
FosteringCourtImprovement.org16
Already Measurable and Measured (AFCARS)
Safety Permanency Timeliness
FosteringCourtImprovement.org17
AFCARS
Adoption and Foster Care Analysis and Reporting System
FosteringCourtImprovement.org18
Uses of AFCARS
Award Adoption Incentive bonuses Allot Chafee Independence $$$ Produce annual reports to Congress
on child welfare outcomes Conduct IV-E eligibility reviews Assess state’s compliance with federal
CFSR performance standards
FosteringCourtImprovement.org19
Platform for Collaboration
Longitudinal AFCARS provides a platform for rolling out the due process & timeliness measures recommended by Building a Better Court
Promotes working partnership between the courts, child welfare agencies and universities,
Provides 9 of the 24 measures defined in Building a Better Court, and 9 of the 30 in ToolKit.
FosteringCourtImprovement.org20
Fostering Court Improvement
What gets measured gets done,
IfYou know it’s being measured.
FosteringCourtImprovement.org21
Stories from Medicine : An Easy Win
Diagnostic Imaging (Cardiac PET/SPECT) Pre-Screen
Very simple questionnaire Predictive Value of the Test
• Surgery: y/n• Angioplasty: y/n
Easy sell to cardiologists Very easy sell to insurance
companies
Bayesian Pre-Screening Tool
FosteringCourtImprovement.org22
Stories from Medicine : A Sea Change
Major resistance, even when shown computer was right more often, lower mortality
Over time (3 years) older docs modified their risk tolerance, less invasive, better outcomes, learned from computer
Tripled business, more patients living longer, much lower costs ...
Automated Reading of Scans
FosteringCourtImprovement.org23
Court Stakeholders Have a Right to be Skeptical
In medicine, I see examples of better outcomes at lower cost quite often.
Have yet to see one in child welfare. Many claims, little evidence Judges, attorneys, CASAs, and caseworkers
have good reason to be skeptical:• Complex Business• Spectacular IT Failures• Terabytes of Unused Data
FosteringCourtImprovement.org24
Get Real Data reports are only meaningful to you in the
context of your job, your reality. We (FCI) have been successful* using data to
develop a shared context among local court stakeholder groups.
Very different jobs, very different realities, but we need to have some common understandings to do successful* work together (and occasionally alter some realities).
*successfully improve an outcome
FosteringCourtImprovement.org25
Diverse Realities: Dependency Court
Caseloads
3,140 US counties + DC
70% of counties remove less than 1 child per week
• 50% less than 1 per 2 weeks
7% of counties (234) average more than 1 child removal per business day
FosteringCourtImprovement.org26
Get Local Much as I’d like to, we
probably can’t do this with just a website.
• our web reports are not for a general audience.
County by county …• Judges convene• Georgia’s CIP buys lunch• I walk through a
Powerpoint of charts & numbers from the website.
FosteringCourtImprovement.org27
Broad Topic Areas of Court Stakeholder Meetings
Removal Rates Reasons for Removal Kids in Care Group Settings IV-E Reimbursement Kids Headed for
Adoption Reunification Rate Time to Adoption
FosteringCourtImprovement.org28
A Simple Temporal Context
Emphasize the Context for every number, nothing without Context.
Temporal: Interpret current measures in the context of the past.
FosteringCourtImprovement.org29
The Context of Comparable Jurisdictions
Emphasize the Context for every number, nothing without Context.
Comparables: Interpret in context of comparable counties.
FosteringCourtImprovement.org30
The Context of Statewide Rank
Emphasize the Context for every number, nothing without Context.
Rank: Interpret in context of state or other ranking.
FosteringCourtImprovement.org31
The Systemic, Process Context
Flow of cases through the system
FosteringCourtImprovement.org32
Outcome Framework Context
Safety• Maltreatment
does not occur.Measures: Proportion of children reported maltreated.
• Maltreatment does not recur.Measures: Proportion of children re-reported within 3, 6, and 12 months; Proportion re-entering foster care.
Permanency• Children remain in
their homes.Measures: Proportion removed from their home.
• Bonded attachments are maintained.Measures: Proportion of siblings placed together; Frequency of visitation with kin; Proportion in a stable placement.
• Permanency is timely.Measures: Proportion reunified with family within 12 months; Proportion adopted within 24 months.
Well Being• Children are
healthy.Measures: Proportion with physical health needs met; Proportion with mental health needs met.
• Children are educated.Measures: Proportion performing at age-appropriate grade level.
Note that these outcomes are from the child’s perspective, not the system’s.
FosteringCourtImprovement.org33
Track Record: Real Work Done in a Shared Context
Douglas County, GA:• Removals cut 50%• Relative placements
increased 10-fold• Adoption rate
doubled Floyd County, GA:
• Removals cut 30%• Pre-teens in group
care down 80%
Hall County, GA:• Adoptions under 24
months tripled Bartow County, GA:
• Removals cut 50% Most counties
unchanged, but actively dissecting their processes
FosteringCourtImprovement.org34
Improving the Product
Additional Reports:• Safety Report from linked NCANDS-AFCARS data• Children Remaining Safe with Family• CFSR 2001, CFSR 2006• Illinois Signs of Progress Reports• Toolkit (9 Measures)
Discovering Best Practices:• Composite Measures• Causal Linkages from Process Measures to
Outcome Measures
FosteringCourtImprovement.org35
Loose Ends Process Measures (caseloads, due process, …) are crucial,
but only when put in the context of Outcome Measures.
Judges care deeply about outcomes. Need bottom-lines to keep us aligned and on task, I
propose:• Children safe with family• Failing that, timely permanency
Shift spending from data acquisition to data analysis. Don’t call GeekSquad™ for data analysis.
(Don’t equate technology and data analysis)
FosteringCourtImprovement.org
Questions, Comments ...
Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results
Andy Barclay, [email protected] and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law
Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006