Welcome!

We will start shortly, but are waiting for people to join, don’t worry if you can’t hear anything yet.

Before we start we will be going through some E-Seminar housekeeping items, so that everyone can participate fully in the online meeting

If you are having difficulties joining the meeting please call Lynsey Ogilvie on 024 7662 7527.

NHS Improving QualityIntroducing Measurement for Improvement

1. Welcome, Introduction & Housekeeping Jeri Hawkins Mental Health & Dementia Delivery Support Manager

2. Introducing Measurement for Improvement Alison Crawford, Measurement & Evaluation Manager

3. Links and Contacts details

AGENDA

NHS Improving QualityIntroducing Measurement for Improvement

www.nhsiq.nhs.uk

Welcome and Introduction

Jeri HawkinsMental Health & Dementia Delivery Support Manager

• During the E-Seminar we will mute all delegate’s lines throughout the presentation.

• If at other times you are in a noisy environment please mute your line by pressing the mute button on your screen (this can be found on the right hand side of the screen)

• If you would like to ask a question please use the raise hand button (this can be found on the right hand side of the screen)

• This is an interactive session, please add your comments, and thoughts into the chat box as we go through the presentation.

• At the Q&A session, type your question into the chat, or raise your hand, we will un-mute all lines during the Q&A.

• If you are having any technical problems, send a message to the Host via the chat panel or call Lynsey Ogilvie on 024 7662 7527.

• We will now start recording this Webinar

Alison CrawfordMeasurement and Evaluation ManagerImprovement Capability NHS Improving Quality

Introducing ‘Measurement for Improvement’

See also http://prezi.com/hjlmbaux8axf/?utm_campaign=share&utm_medium=copy&rc=ex0share

“You can’t fatten a cow by weighing it”

(Palestinian proverb)

However, how else will a farmer know when to send a cow to market unless he measures it?

What is ‘Measurement for Improvement’?

Measurement - the size, length, or amount of something, as established by measuring

Improvement - a thing that makes something better or is better than something else

So measurement for improvement is the process we go through to measure the things that we are trying to improve, so that after we’ve made a change we can demonstrate that’s it worked

The next hour…..

This session will……• Work through a simple example of using data to show if a new

intervention works• Practical considerations for real life situations• Where to go next for more information and tools

This session won’t………..• Delve into the theory of the statistics behind measurement for

improvement• Cover capacity and demand

So why is a little bit scary…..?

• Target driven environments, which may encourage counterproductive behaviour

• Endless submission of data, forms and measures into a black hole

• The language is gobbledegook• Something that ‘analysts’ or other people do• Does it use complicated maths? • Clinical trials need HUGE samples of patients• Endless data collections, hard to find the right information• Data is out of date

The good news….

Measuring things for improvement is probably one of the simpler and most meaningful types of measurement!

• Doesn’t need large sample sizes (works on a ‘just enough’ principle)

• You choose the measure that’s relevant to the thing you are trying to improve (no externally decided measures)

• Can be as simple as a count or percentage• If you choose to collect your own data, it can be as up-to-date as

you want it to be• Can be done with a simple line chart over time (but there are

more complex things too if you want to be extra clever)

Questions to ask

1. What is the aim of this improvement exercise? Can I distil this into a 2-minute elevator pitch?

2. What exactly is the problem and what is the size of it?

3. What sort of changes are ‘normal’ and how will we know if we’ve made things better?

4. What does success look like, do we have a specific target?

Gather data to understand the problem

Existing data• National data collection• Clinical audits• Local patient administration systems• Risk Management Systems• Financial Systems• Surveys / samples of patient records

New data • Surveys• New audit or data collection

Choosing measures

Choose measures based on data you have available

Work out your ideal measure, and then find data which fits

• Great if you know the data well

• Might skip to a ‘proxy’ measure without looking for a better fit data

• More thorough approach to exploring the best measures

• Doesn’t assume you know the data well

• Time consuming, needs follow up research

Driver diagram

Aim or Objective

Primary Driver 1

Secondary Driver 1

Primary Driver 2

Secondary Driver 2

Secondary Driver 3

Secondary Driver 4

Driver diagram

Lose 2 stone in weight in 6

Months

Healthy eating

Three calorie controlled meals per day

Motivation

Fruit and veg snacks only

Plan for social eating

Limit alcohol

Daily exercise 20 mins per day

Exercise

Rewards for milestones

Weekly weigh in

Use pedometer to measure steps

Weight

Calories consumed

Units consumed

Steps per dayDistance travelled

Calories burned

BMIWaist

circumference

Treats consumed

What does the raw data look like?

So we have data…..

5 people in Leeds West CCG were treated by the Early Intervention Team in April 2014

But what does that mean?!!

• Is 5 a good or a bad number?• Is this higher or lower than other similar CCGs?• Is it normal for this number to fluctuate each month? • Is there a number which represents ‘success’? (ie which the CEO

will be happy with, and you can say patients who need support are getting it?)

0 1000 2000 3000 4000 5000 6000 7000 80000.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

Nationally 1.4% of people saw an EITIn Leeds West this was 0.1%, which was low compared to other CCGs across the country

Therefore we might infer that 5 people seeing an EIT is too low, and could be improved

However, we might also check local information about why that number might be low, such as problems with this data source or alternative ways people are being supported that don’t fit this definition.

So what next?

Based on your local intelligence, you need to plan in some improvement activities, and implement them one by one to see if they make the situation better.

That is, a sustained improvement which doesn’t appear to be down to chance or normal variation in the data.

You can then do something which makes the ‘improvement’ permanent or part of everyday business (and look for other things to improve…)

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber0

2

4

6

8

10

12

14

16

18

20

Patie

nts v

isiti

ng E

IT

Run charts – where are we now?

• What’s your baseline, is it steady?

• Does it reflect what frontline staff think is happening?

• Is it a reliable source of data – ie changes in the numbers not due to problems with data collection, such as who is on a shift, changes in definition and so on.

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber0

2

4

6

8

10

12

14

16

18

20

Patie

nts v

isiti

ng E

IT

Run charts – what will we be doing?

Referral pathway to EIT mapped and revised to be more efficient

Email and twitter campaign to all staff

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber0

2

4

6

8

10

12

14

16

18

20

Patie

nts v

isiti

ng E

IT

Run charts – where do we want to be?

Referral pathway to EIT mapped and revised to be more efficient

Email and twitter campaign to all staffThe National level is 1.4% of people treated by mental health

services saw an Early Intervention Team

Assuming the same proportion for Leeds West CCG6090 x 1.4% = 85 referrals per month

Is 85 achievable, given the existing maximum of 5? Is there capacity in the EIT to see 80 extra people in a month?

Do these planned actions expect to increase referrals to this level?What does the CEO expect to achieve?

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber

January

Febru

ary

Marc

hApril

May

June

July

August

September

October

November

Decem

ber0

2

4

6

8

10

12

14

16

18

20

Patie

nts v

isiti

ng E

IT

Run charts – what can we measure?

Referral pathway to EIT mapped and revised to be more efficient

Email and twitter campaign to all staff

There might not always be a relevant source of data published

1. Check with your local information team about sources of local data they can recommend

2. Consider asking someone to add in collecting the data you need to an existing collection

3. If you do need something new, make sure you get advice on new surveys to make sure its valid and as easy as possible

Top tips for new surveys or audits

• Keep it short and sweet • Survey questions, consider testing on a couple of people to check

they are understood and make sense• Use a fixed way of collecting, such as a form or spreadsheet• Write down any definitions you use, and share with people doing

the measuring• If you are using equipment to measure, make sure it’s serviced

and calibrated to remove bias• CAUTION – if you collect patient names get some advice on

storing and publishing the results

• Once you have determined that your interventions have worked, STOP collecting data

Run charts – quick lesson in variation

November

December

January

Febru

aryMarc

hApril May

JuneJuly

August

Septem

ber

October

November

December

January

Febru

aryMarc

hApril May

JuneJuly

August

Septem

ber

October

November

December

0

2

4

6

8

10

12

14

16

18

20

Patie

nts v

isitin

g EI

T

Intervention1

A shift: six or more consecutive data points either all above or below the median. Points on the median do not count towards or break a shift.A trend: five or more consecutive data points that are either all increasing or decreasing in value. If two points are the same value ignore one when counting.

Baseline

MEDIAN = 4

Has this intervention made an improvement?

Things that show normal variation –• Temperature outside changes , during a day, a season or a year• Number of visitors to a shop, by time of the day• Number of cars driving on stretch of road, by time of day• Number of crisps in a packet• Height of men and women• The time it takes to walk 100m

Known as common cause, this is the natural variation that we experience day to day

Things that show unusual variation – • Temperature inside a home which has central heating or air conditioning• Traffic on the road during le Tour de France• Number of crisps in a bumper size packet of crisps (compared to a normal size packet)• The time it takes to cycle 100m (compared to walking 100m)

Known as special cause, this is where something special or new has changed the normal passage of events. In improvement we are looking for a positive ‘special cause’, much like adding central heating to a house or increasing the size of a crisp packet

Run charts – lets look at some data

November

December

January

Febru

aryMarc

hApril May

JuneJuly

August

Septem

ber

October

November

December

January

Febru

aryMarc

hApril May

JuneJuly

August

Septem

ber

October

November

December

0

2

4

6

8

10

12

14

16

18

20

Patie

nts v

isitin

g EI

T

Intervention1

A shift: six or more consecutive data points either all above or below the median. Points on the median do not count towards or break a shift.A trend: five or more consecutive data points that are either all increasing or decreasing in value. If two points are the same value ignore one when counting.

Baseline

MEDIAN = 4

Has this intervention made an improvement?

November

December

January

Febru

aryMarc

hApril May

JuneJuly

August

Septem

ber

October

November

December

January

Febru

aryMarc

hApril May

JuneJuly

August

Septem

ber

October

November

December

0

5

10

15

20

25

30

Patie

nts v

isitin

g EI

T

Baseline

Intervention2

Intervention1

MEDIAN = 5

A shift: six or more consecutive data points either all above or below the median. Points on the median do not count towards or break a shift.A trend: five or more consecutive data points that are either all increasing or decreasing in value. If two points are the same value ignore one when counting.

More complex improvement challenges

• Politically sensitive• Mortality• Children’s services• Vulnerable patients/clients

• Intangible and difficult to capture• Social return on investment• Experience of care• Patient dignity• Compassionate care

• Complex measures or indicators• Mortality – Standardised Hospital

Mortality Indicator vs Hospital Standardised Mortality Rate

What are your challenges?

Addressing challenges

• Accept no measure is perfect, it shows a snapshot of a situation. You will need more than one measure or source of information to tell the whole story

• Measure activity and new ‘stuff’ created as well as outcomes – this will help show short term progress

• Involve users/stakeholders in choosing the desired outcomes and setting the key measures of success

• Some measures are highly technical, just because the thing they measure is complicated. Get specialist advice if you need it, also compare against simpler measures (such as no of deaths)

• Use softer data, such as focus groups etc if needed. It all contributes to telling the story!

Summary

1. Using measurement as part of your improvement work is valuable and can be kept simple

2. Understand your existing or baseline data – does it represent the real world you see? Ask questions

3. Capture data over time, to look for a shift or a trend which tells you that something unusual is happening (hopefully your planned improvement!)

4. Get help if you need it – analysts will find this sort of work very interesting

Useful resourcesInstitute for Healthcare Improvement – whiteboard series with Dr Bob Lloyd, talking through the theory and tools of improvementwww.ihi.org

Quality Improvement Scotland – great eLearning tool, which includes measurement for improvement moduleswww.qihub.scot.nhs.uk/education-and-learning/qi-e-learning.aspx

NHS Institute – many products still available and relevant, such as the good indicator guide and Mike Davidge’s videos (NB. The availability of this resource at this web address may be time limited)www.institute.nhs.uk

NHS IQ - let us know if you’d like a masterclass in a specific topic and we’ll host something or create something to helpHow to become an Improvement Measure Expert in 90 minutes – 12 activities which will get you on the path to hands-on measurement http://prezi.com/hjlmbaux8axf/?utm_campaign=share&utm_medium=copy&rc=ex0share

CLOSETHANK YOU FOR JOINING US TODAY

Any questions?A link to the presentation will be sent to you via email.Please address additional questions or comments to:• Jeri.hawkins@nhsiq.nhs.uk• Alison.crawford@nhsiq.nhs.uk