Building Local Capacity for Data Analysis and Use

Sharon Walsh, Data Accountability Center (DAC)

Mary Anne White, Infant & Toddler Connection of VA

Haidee Bernstein, Data Accountability Center (DAC)

Beverly Crouse, Infant & Toddler Connection of VA

Data Accountability CenterGoal

Form partnerships in states that join state and local agencies in the use

of data to drive improved results

Premises

Data Use Involves:• Working through a Collaborative

Team approach.• Engaging Team in a Continuous

Improvement Process.• Relating the Data to specific

Problem/Issue.

Using Data is an Iterative Process!

There is a Process for Using Data to

Improve Performance!

Important Points for Helping Local Data teams

Be SuccessfulHow do you maximize data you

already collect and collect what you need?

How do you organize your staff and your agency around ongoing data use?

Its all about continuous improvement Use data to determine priority for

focusIt is important to “drill down” to

understand performance to identify meaningful solutions

1. Identify relevant data

2. Conduct data analysis to generate hypothesis3. Test

Hypothesis to determine root cause

4. Plan for Improvement

5. Evaluate Progress

Data Analytics

Inquiry

Act

ion

DATA ACCOUNTABILITY CENTERDATA ANALYTICS

Historical PerspectiveVirginia’s 2008 Determination

Status

Work with DAC

Leadership Academy (April

2010)

Data Analysis Modules

Historical Perspective

Historical Perspective

Leadership Academy April 2010Two Sessions Held: Plenary & Breakout

Plenary: overview of use of quality of data

Breakout sessions: Use of actual local data

Results:Positive feedback from meeting evaluationsWanted more time to spend on the activityFirst activity in all CAP or SEP’s developed

requires a data analysis be completed

Historical Perspective

Standards and Principles

Use data on a regular

basis

Use data for continuous

improvement

Verify the accuracy of your data

Make sure you have the right team at each

step

Own your data

Use a process to determine how much data is

needed

Haidee’s stuffI

n

D

e

c

e

m

b

e

r

2

0

1

0

,

w

o

r

k

o

n

t

h

e

P

o

w

e

r

P

o

i

n

t

p

r

e

s

e

n

t

a

ti

o

n

b

e

g

a

n

In February 2011, The PowerPoint slides were put onto SharePoint

Today, we are officially showing some of the slides and explaining the process and rationale

From Notes to PowerPoint

From PowerPoint to Lectora

PowerPoint Lectora Final Product

• Why engage in this process• Types of Data

Module 1: Overview

• Identify people to look at the data• Define the problem(s)

Module 2: Preparation

• Identify Relevant Data• Conduct Data Analyses to Generate

Hypotheses• Consider and test Hypotheses

Module 3: Inquiry

• Determine Actionable Causes• Develop and Implement Improvement Plans• Evaluate Progress

Module 4: Action

• Case StudyModule 5: Practice

Guide

Infant & Toddler Connection of1 Alexandria 11 Danville-Pittsylvania 21 the Highlands 31 Prince William, Manassas and Manassas Park2 the Alleghany Highlands 12 Dickenson 22 Loudoun 32 Rappahannock-Rapidan3 Arlington 13 Crater District 23 Middle Peninsula-N Neck 33 Rappahannock Area4 the Roanoke Valley 14 the Eastern Shore 24 Mount Rogers 34 the Blue Ridge5 Central Virginia 15 Fairfax-Falls Church 25 the New River Valley 35 Richmond6 Chesapeake 16 Goochland-Powhatan 26 Norfolk 36 the Rockbridge Area7 Chesterfield 17 Hampton-Newport News 27 Shenandoah Valley 37 Southside8 Williamsburg * James City * York * Poquouson 18 Hanover 28 the Piedmont 38 Valley9 Planning District 14 19 Harrisonburg-Rockingham 29 LENOWISCO

39 Virginia Beach10 Cumberland Mountain 20 Henrico-Charles City-New Kent 30 Portsmouth 40 Western Tidewater

Local Lead Agencies

Infant & Toddler Connection of1 Alexandria 11 Danville-Pittsylvania 21 the Highlands 31 Prince William, Manassas and Manassas Park2 the Alleghany Highlands 12 Dickenson 22 Loudoun 32 Rappahannock-Rapidan3 Arlington 13 Crater District 23 Middle Peninsula-N Neck 33 Rappahannock Area4 the Roanoke Valley 14 the Eastern Shore 24 Mount Rogers 34 the Blue Ridge5 Central Virginia 15 Fairfax-Falls Church 25 the New River Valley 35 Richmond6 Chesapeake 16 Goochland-Powhatan 26 Norfolk 36 the Rockbridge Area7 Chesterfield 17 Hampton-Newport News 27 Shenandoah Valley 37 Southside8 Williamsburg * James City * York * Poquouson 18 Hanover 28 the Piedmont 38 Valley9 Planning District 14 19 Harrisonburg-Rockingham 29 LENOWISCO

39 Virginia Beach10 Cumberland Mountain 20 Henrico-Charles City-New Kent 30 Portsmouth 40 Western Tidewater

Local Lead Agencies

Ways to Use DataIdentifying issues

Monitoring

System Planning

Improvement Activities

System oversight/man

agement

Approach to Improvement Planning

Monitorin

g Consultants

TA consultants

Local

System

Planned preparation

Possible Reactions

Negative Reactions

• Potential Roadblocks– I do not have time

for this– I already know this– I know the problems– I have the solutions

Positive Reactions

• Potential Facilitators– In the long run this

will save time– I didn’t know this

was possible – This information will

help me do my job better

– This information will help families

Proactive Versus ReactiveBoth are Positive

Check the data to ensure its accuracy

Determine program effectiveness. Develop a plan using local data

Proactive

Use available data to respond to a problem

Adjust plans that are in place after conducting data analysis

Reactive

What is Your Purpose

Reactive• Example: Responding

to an issue such as monitoring results

• Purpose: To address monitoring results that are below the state target

Proactive• Example: Conduct

quality review or assessment to determine areas of need

• Purpose: To Proactive look at the quality of dataGood Idea Good Idea

How Will Your Team Interact?

Who?

Can decipher the data?

Can bring new ideas?

Can make the

decisions?

Can translate data into policy?

Pre On-Site Visit

With Local System ManagersDiscuss purpose of data analysis processDiscuss potential data team members Identify ITOTS reports to be reviewed Identify data from other sources that need to be reviewed

Pull three years worth of data

Desk AuditReview and analyze same data as local systemFormulate questions about data Identify additional data that may need to be collected

First On-Site Visit

Inq

uir

yP

rep

arat

ion

3 . Identify Relevant Data

4 . Conduct Data Analysis to Generate Hypotheses

5 . Test Hypotheses to Determine Actionable Causes

1. Define and Articulate the Problem

2. Define the Problem/Issue

Beginning the Journey

1. Complete the Preparation Phase and part of Inquiry Phase

2. Review the data reports “What does the data tell you?”

What are the good things the data is telling you? What surprises you about the data? What questions strike you as you look at the data? What data appears to be missing?

What are the good things the data is telling you? What data appears to be missing?

Review Multiple Source of Data

Data from interviews with families, service coordinators, EC community partners, etc.

Record Reviews

Part C, Local and State data systems

During Inquiry Phase

Infant & Toddler Connection of Playground CityReferral Outcome by Referral Source

7/1/09 – 7/30/10

Referral Sources Evaluated Not Evaluated

Total Referral Source

Eval-Ineligible

Will Receive Services Total Unable to

ContactDeclined Screening

Declined Eval Total

Health 9 9 3 1 3 7 16

DSS 1 5 6 2 2 5 9 15

Doctor’s 1 10 11 4 5 8 17 28

Parent 3 10 13 1 6 7 20

Other 1 8 1 1 1 3 12

Totals 7 49 56 12 10 28 50 106

Infant and Toddler Connection of Playground CityReferral Outcome by Referral Source (7/01/09-

7/30/10)1. Information Local System

Gathered through this report:53% of all referrals are

evaluated; 47% are not evaluated– 46% of all referrals will

receive services– 6% of all referrals were

evaluated ineligible– 11% of all referrals were lost

to contracts– 9% of all referrals declined

screening26% of all referrals declined an

evaluation

2. Physician Referrals: 26% of all referrals

3. Parent Referrals: 19% of all referrals

4. Health: 15% of all referrals5. Dept. of Social Services: 14% of

all referrals

A. Physician Referrals: 39% were evaluated; 69% were not evaluated. Of those not evaluated, 39% declined either screening or evaluation.

B. Family Referrals: 65% were evaluated; 35% were not evaluated. Of those not evaluated, 100% declined screening or evaluation.

C. Health Referrals: 56% were evaluated; 44% were not evaluated. Of those not evaluated, 43% were lost to contact and 57% declined screening or evaluation

D. DSS Referrals: 40% were evaluated; 60% were not evaluated. Of those not evaluated, 22% were lost to contact and 78% declined screening or evaluation

Additional Data Needed

• What is the average age of referrals?• Which physicians are referring?

Specific name versus name of practiceWhat is the average age of the physician referral?

• How do families hear about Part C services?• Why are families declining Part C services?

At what point in the process are families declining Part C services?

Second On-Site Visit

Inq

uir

y

3. Identify Relevant Data

4. Conduct Data Analysis to Generate Hypotheses

5. Test Hypotheses to Determine Actionable Causes

What’s Accomplished?

Summarize discussion from previous visit

Review data collected in-between visits

Formulate hypotheses• H

ypothesis is a proposition or supposition tentatively accepted to explain certain facts or to provide a basis for further investigation.

Identify strategies to test hypotheses

Data Collection (8/1/10 – 11/30/10)

• Average age of referral: 16 months• 10 referrals received from

physicians:– Average age of referral: 14 months

• Dr. Swingset: 0 referrals received < 18 months

• Dr. Sandbox: 0 referrals received < than 24 months

• Dr. Bottle: average age of referral is 9 months; 50% of declined a screening

• No referrals from the NICU at the ABC hospital

Data Collection (8/1/10 – 11/30/10)

• 12 referrals received from family’s:– 7 Families declined services:

• 57% of families felt their child was developing at age level

• 43% of families wanted to receive services through a private agency

• 5 families declined a developmental screening

• 2 families declined Assessment for Service Planning

ITC Playground City Hypotheses

Physicians are not referring children at very young ages

Physicians are not providing families with a complete explanation of early intervention and reason for referral

Hospitals are not referring premature babies

Final On-Site Visit

Act

ion

6. Develop and Implement Improvement Plan

7. Evaluate Progress

Inq

uir

y

3. Identify Relevant Data

4. Conduct Data Analysis to Generate Hypotheses

5. Test Hypotheses to Determine Actionable Causes

Final On-Site Visit

Moving from inquiry to action

Review Hypotheses• Were we correct?• Do we need to re-look at data to formulate new or additional hypotheses?

Improvement Planning and Evaluating Progress• Consider priorities• Sphere of Influence• Use data to determine if moving in right direction

ITC Playground City Improvement Plan

Plan to address increase in referrals of premature babies from NICU: Identify Discharge Social Workers, Nurses or

Therapists responsible for referralsMeet with individualsGather data from hospital (# of premature births

residing in their community, where are referrals being made)

Provide information about EI in VirginiaCollaboratively develop mechanism to meet with

family prior to NICU discharge

What’s Next?

•Develop mechanism to introduce to local systems

Complete data analysis modules

•Analysis work completed with systems to-date

Fine-tune process work with local

systems

•Competing prioritiesDevelop mechanisms

on how to keep people engaged in

this process

Things to Remember

States can assist local agencies/programs to remember:

It is all about improved quality of services for children and families

Hard to let go of traditional improvement planning

Hard to let go of your own sense of what the problem/solution is

Follow the data where it leads you Ask the difficult questions Create an environment where solutions are

generated