Measurement for Improvement

Turn to your neighbor• What have been your biggest learnings or

challenges regarding data gathering and measurement for your Early Years work?– Definitions– Collection– Reporting– Frequency– Analysis– Other?

Components of a Learning System 1. System level measures2. Explicit theory or rationale for system changes3. Segmentation of the population4. Learn by testing changes sequentially5. Use informative cases: “Act for the individual learn for

the population” 6. Learning during scale-up and spread with a production

plan to go to scale7. Periodic review8. People to manage and oversee the learning systemFrom Tom Nolan PhD, IHI

What are we trying toAccomplish?

How will we know that achange is an improvement?

What change can we make that will result in improvement?

The Model for Improvement

Act Plan

Study DoSource:

Langley, et al. The Improvement Guide, 1996.

Our focus today

The three questions

provide the strategy

The PDSA cycle provides the tactical approach to work

AIM (How good? By when?)

Concept Measures

Operational Definitions Data Collection Plan Data Collection

Analysis PDSA

The Quality Measurement Journey

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

Case Study• Promoting Bedtime Reading• Lochrin and Grassmarket Nursery Schools• Edinburgh• Lead: Donna Murray

AIM – Improve bedtime routines, attachment, literacy

Concept – Increase bedtime reading

Measures – Percentage of children receiving a bedtime story

Operational Definitions – Number read story/Total Children

Data Collection Plan – daily; no sampling

Data Collection – teacher asks parents at drop off, spreadsheet

Analysis – run chart PDSA


Children have all the developmental skills and abilities expected at the start of primary school

Societal Issues

Child’s physical & mental health and emotional development

Improved sharing of information

Improved management, planning and quality of services

Improved joint working

Improved identification

Detailed Aim:90% of all children within each CPP have reached all of the expected developmental milestones at the time the child starts primary school, by end-2017

Health

Improved child’s dental health

Improving child nutrition

Attachment

Improved stability / permanence for LAC

Improved uptake of benefits

Improved family centred response

Improved leadership, culture & planning`

Improved teamwork, communication and collaboration

Improving brain development and physical play

Early Learning & Play

WORKSTREAM 3 (30 months to start of primary school)

Identification & reasons for current resilience

Aim 1⁰ 2⁰

Theory of what drives developmental milestones

Theory of what actions will ensure developmental

milestones are reached at the start of primary school

Poverty

Quality Of Home Environment

Domestic Abuse & Violence

Workforce Issues

Transport, Community Capacity & Cultures

Access To Services

Employment

Carer’s physical & mental health and skills

Level of education

Misuse of alcohol & drugs

Nutrition

Disabilities & Mental health

Parenting skills & knowledge

Additional Support

Version: 06/03/2013


Concept Measures


Analysis PDSA



Project Aim Statements

• What, by when, by how much.–“By the end of June, 90% of

children at Grassmarket Nursery will receive a bedtime story at least 3 times a week.”


Concept Measures


Analysis


PDSA


Children have all the developmental skills and abilities expected at the start of primary school

Societal Issues

Child’s physical & mental health and emotional development

Improved sharing of information

Improved management, planning and quality of services

Improved joint working

Improved identification

Detailed Aim:90% of all children within each CPP have reached all of the expected developmental milestones at the time the child starts primary school, by end-2017

Health

Improved child’s dental health

Improving child nutrition

Attachment

Improved stability / permanence for LAC

Improved uptake of benefits

Improved family centred response

Improved leadership, culture & planning`

Improved teamwork, communication and collaboration

Improving brain development and physical play

Early Learning & Play

WORKSTREAM 3 (30 months to start of primary school)

Identification & reasons for current resilience

Aim 1⁰ 2⁰

Theory of what drives developmental milestones

Theory of what actions will ensure developmental

milestones are reached at the start of primary school

Poverty

Quality Of Home Environment

Domestic Abuse & Violence

Workforce Issues

Transport, Community Capacity & Cultures

Access To Services

Employment

Carer’s physical & mental health and skills

Level of education

Misuse of alcohol & drugs

Nutrition

Disabilities & Mental health

Parenting skills & knowledge

Additional Support

Version: 06/03/2013

Drivers

Drivers

Drivers

Concept:Bedtime Reading

Driver Concept and Measure

• Measures to provide feedback on the concept– % receiving a story– % of stories read at bedtime– % of parents reporting improved bedtime routine– % enjoying the bedtime story– % reporting increase in bedtime story reading

AIM (Why are you measuring?)

Concept Measures


Analysis PDSA



Operational Definitions• What does reading a story mean?• When is bedtime versus other time?• What is improved bedtime routine?• What is the definition of enjoyed?


Concept Measures


Analysis PDSA



Data Collection Planning & Doing

• All children are included, no sampling. 7 days a week.

• Donna will inquire with each parent at drop off Monday through Friday.

• Data will be captured on a paper form. • Donna inputs daily data into spreadsheet

following drop off.


Concept Measures


Analysis PDSA



Analysis – Run ChartsChildren receiving a bedtime story

MedianGoal

0

20

40

60

80

100

120

M T W TH F M T W TH F M T W TH F M T W TH FDay of the week

Perc

enta

ge

of c

hild

ren

.

Parents survey

Grassmarket changes

introduced.

Books available at collection time.

Research information handed to

parents.

weekly average displayed for parents


Concept Measures


Analysis PDSA



Action – Testing Change Ideas

NHS Lanarkshire – Our 27-30 Month Review Data

Journey

Stacey Rooney

Background• NHS Lanarkshire Clinical Quality Service,

supporting both North and South Lanarkshire CPPs from a health perspective

• 27-30 month reviews commenced June 2013 in Lanarkshire – measurement data for stretch aim 2

• But we don’t have baseline data…….• ….so we need early access to our review data to

develop a baseline…….• ISD Proposed quarterly reports…….November

2013 before we see our first set of results

Accessing our data – lots of questions....

• Where do the forms go?• Does anyone in Lanarkshire have access to the

CHSP system?• Can we access the system?• Technical challenges accessing the

system....patience.......patience........we’re in!!• We’re in but what do all these codes mean?• Expertise to interrogate the data

How did we access CHSP Business Objects Universe (AKA – our data)?

• Contact the ATOS Helpdesk [email protected]

• Receive a form to complete via email, needs authorising by your Child Health Administrator

• Receive complex instructions from ATOS on how to set-up your PC/laptop – stick with it, you’re getting there!!

• Receive your Business Objects Universe login and password from your local Child Health Administrator

mailto:[email protected]

CHSP Business Objects

Benefits of accessing our data

• Earlier access to our data – some delays come from the time to return the forms for data entry and data entry itself (approx 4 weeks)

• Early identification of quality issues with form completion enabled us to feed this back and engage with staff

• We have early baseline data• Stratify the data as we like (unit, locality, team)• Results are helping us identify early tests of

change

Early data focus• Meaningful reviews (9 developmental milestones

reviewed and recorded)• Stretch aim 2 (meeting all developmental

milestones)• Future actions (request for assistance/referral)• Looked After Status• Breeches• Health Plan Indicator (HPI) – status change• Attendance / DNA – obtained locally

Coverage – A work in progress:

Number of Eligible Children

Number Invited for Review

Number Attending for Review

Number of Meaningful

Reviews

A national definition of ‘eligible children’ in a reporting period is still in development.

Eligible = denominator/starting pointInvited: coverage measure 1Attended: coverage measure 2Meaningful: coverage measure 3

Coverage: An early example

July August September0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

27-30 Month Review - Attendance July-Sep 13 (NHSL 1 Unit)

Locality 1Locality 2Locality 3

Examples of our early data

Jun-13 Jul-13 Aug-13 Sep-13 (p)0

50

100

150

200

250

300

350

400

450

257

309

392

289

NHSL Number of Reviews - Jun to Sep 2013

Lanarkshire

Month

No. o

f rev

iew

s

(p) Provisional data - incomplete

Jun-13 Jul-13 Aug-13 Sep-13 (p)0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

76% 76%72% 74%

% of Meaningful Reviews - Jun to Sep 2013

Meaningful Reviews %

Month

%


Jun-13 Jul-13 Aug-13 Sep-13 (p)0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

22.39

17.15

23.72

29.76

Future Actions - Rate per 100 reviews Jun to Sep 2013

Lanarkshire

Month

Rate

per

100

revi

ews


Jun-13 Jul-13 Aug-13 Sep-13 (p)0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

85% 87% 86%79%

No Concerns Identified (Stretch aim 2) - Jun to Sep 2013

No Concerns %

Month

%


Limitations of our early data• We’re still developing a robust process to

measure uptake - so results are only representative of those attending

• Non attenders are potentially our more vulnerable children and this may be skewing our results – particularly stretch aim 2

• Non-meaningful reviews are being excluded, we need to reduce this (approx 25%)

Stage 2 data focus• What other data is available from

the review?– Age of child at review– SIMD quintiles– Ethnicity– Place of review (home, GP practice,

clinic)– Exposure to smoke– Registered with a dentist– Bi/Multi-lingual

Reporting

Monthly reports

Monthly summary and locality level reports are in development

Next steps• Continue to work with colleagues to develop reporting

that is meaningful and facilitates/guides improvement• Continue to feedback real time issues with data

completeness and quality• Further work to link other reviews and develop data

intelligence over time for each child e.g. HPI• Further work to consider how best to share this data

with partners out with health to benefit children and families

Thank you. I hope that didn’t leave you feeling like this...................or this.......

I hope it did make you feel like this.....

Thank you to Scott Purdie, Clinical Quality Co-ordinator, NHS Lanarkshire for his work in this area.

And

My niece Emily, age 9 months for brightening up the data talk!

[email protected]

mailto:[email protected]

Measurement for Improvement

Documents

Transcript of Measurement for Improvement