BCF Scheme Measurement and Evaluation Workshop · PDF file 1 The Better Care Fund BCF Scheme...

www.england.nhs.uk 1

The Better Care Fund

BCF Scheme Measurement and

Evaluation Workshop

(NHS England in Partnership with London

ADASS)


Neighbourhood Rm, Coin Street Conference Centre, 108 Stamford St, South Bank, London, SE1 9NH

Tuesday 25th April 2017

@ 9 – 13:30 pm

BCF Schemes Metric and Evaluation Workshop

1. Registration and Refreshments 9:00 – 9:30 am

2. Welcome, Introductions and Regional Context - Discussion 9:30 – 9:55 am Eileen Sutton & Jane Hannon - NHS England

3. Nuffield Report and Evaluating Shifting the Balance of Care – Discussion and Q & A 9:55 – 10:35 am Natasha Curry - Nuffield Trust

4. Practical Lessons and Tips from Evaluating NHS New Care Models and Vanguards – Discussion and Q & A 10:50 - 11:30 am

Michael Lawrie - NHS England

5. Developing an evaluation framework to support areas in assessing the impact of their schemes – - Discussion and Q & A Anne Jarrett & Fiona Russell - Local Partnerships 11:30 – 12:10 pm

6. Measurement and analysis for Improvement – - Discussion and Q & A 12:40 – 13:20 pm Susanna Shouls - NHS Elect

7. Round up 13:20 – 13:30 pm

Break

Lunch

www.england.nhs.uk

BCF Schemes

Metric and

Evaluation

Workshop

www.england.nhs.uk

Context

4

• Local BCF leads have highlighted evaluating the impact of BCF schemes as a challenge

• Good practice regionally includes :

o Dashboards monitoring overall performance (Camden and Southwark have shared their work)

o Tracking outcomes for cohorts using NHS number to identify patients

o Measuring outputs and outcomes for individual schemes, including patient satisfaction

o Monitoring overall HWB area outcomes and patient satisfaction levels

o Funded external evaluations

• HWB areas have been grappling with what reasonable assumptions are when linking

scheme outcomes and outputs with systems level outcomes

• There is more expectation than ever on the BCF schemes to support system change and

there is an appetite to form a clearer picture of “what good looks like”

• Today we’ve brought together some experts to help us think about these questions.

• We’ll be asking you to share learning, views and ideas via post-it notes on the four flipcharts

• We’d like to start by asking you to take 10 minutes identify in your tables at least one

element in this area that is working well and an area of challenge



Natasha Curry

Senior Fellow in Health Policy

Nuffield Trust

Natasha Curry, Senior Fellow in Health Policy, Nuffield Trust

Shifting the balance of

care

Today’s presentation

7

1. Shifting care out of hospital: Why do it?

2. Community-based initiatives: What does the evidence say?

3. Shifting care out of hospital: why is it so hard?

4. Top tips for implementation and evaluation

5. Q&A

Shifting care out of hospital: why do

it?

8

The Triple Aim:

1. It’s cheaper

2. It improves quality of patient care

3. It improves health and wellbeing

Multiple policy interventions, little

shift

9

2000 2006 2013 2014

“Shift in the centre

of gravity of

spending.”

“Significant

expansion of care in

community settings.”

“Out-of-hospital care

needs to become a

much larger part of

what the NHS does.”

“Ease the pressure

on hospitals.”

But now there’s extra imperative

Trends in hospital activity 10

Financial pressures

Reference: Robertson et al (2017) The King’s Fund 11

£22bn gap by

2020

STP assumptions to bend the

demand curve & save money

12

Area No. of

STPs

Min

Reduction

Max

Reduction

Average by

2020/21

Outpatients 19 7% 30% 15.5%

Elective Inpatients 22 1.4% 16% 9.6%

A&E attendances 26 6% 30% 17%

Non-elective

inpatients

30 3% 30% 15.7%

What do systems leaders think?

13

Care in the community is cheaper and provides better

care for patients. 38%

Care in the community provides better care for

patients but is not cheaper. 38%

Care in the community is cheaper but does not provide better care for

patients. 3%

Care in the community is neither cheaper nor

provides better care for patients.

7%

I am not sure. 14%

With regard to moving care out of hospitals, which of the following statements most accurately reflects your view?

(n=58)

What does the evidence say?

What we did

15

• Reviewed evidence for 27 common initiatives intended to reduce hospital utilisation

• High quality evidence (Cochrane, RCT, systematic reviews) with ‘grey’ literature where other evidence not available

• Focused on cost savings but recognised other values

Categorised initiatives into:

1. Positive evidence re reduced costs/activity

2. Mixed or emerging evidence re reduced costs/activity

3. Evidence to suggest increased costs/activity

The state of the evidence

16

• Limited

• Small studies

• Many are poorly-constructed

• Most are single-disease focused

• Few focus on cost

• Few consider the whole system

• Context-specific

• But, many demonstrate positive impacts in terms of patient experience or outcomes

Evidence suggests some initiatives may

reduce activity and save money

Most Positive Emerging positive

Remote monitoring of people with certain

LTCs

Patients experiencing GP

continuity of care

Improved end-of-life care in the

community

Extensivist model of care for

high risk patients

Condition specific rehabilitation Social prescribing

Targeted support for self care Senior assessment in A&E

Additional clinical support to people in

nursing and care homes

Rapid access clinics for urgent

specialist assessment

Improved GP access to specialist

expertise

Ambulance/paramedic triage to the

community

Evidence suggests some initiatives may

reduce activity and save money

Most Positive Emerging positive

Remote monitoring of people with

certain LTCs

Patients experiencing GP continuity

of care

Improved end-of-life care in the

community

Extensivist model of care for high

risk patients

Condition specific rehabilitation Social prescribing

Targeted support for self care Senior assessment in A&E

Additional clinical support to people

in nursing and care homes

Rapid access clinics for urgent

specialist assessment

Improved GP access to specialist

expertise

Ambulance/paramedic triage to the

community

Many initiatives may not save or may cost money

Mixed – re £ + activity May cost ££

Case management and care

coordination

Extending GP opening hours

Intermediate care: rapid response

services

Specialist support from a GP with a

special interest

Intermediate care: bed-based

services

Consultant clinics in the community

Hospital at Home NHS 111

Shared care models for the

management of chronic disease

Urgent care centres including minor

injury units (not co-located with A&E)

Virtual ward Referral management centres

Shared decision making to support

treatment choices

Direct access to diagnostics for GPs

Many initiatives may not save or may cost money

Mixed – re £ + activity May cost ££

Case management and care

coordination

Extending GP opening hours

Intermediate care: rapid response

services

Specialist support from a GP with a

special interest

Intermediate care: bed-based

services

Consultant clinics in the community

Hospital at Home NHS 111

Shared care models for the

management of chronic disease

Urgent care centers including minor

injury units (not co-located with A&E)

Virtual ward Referral management centers

Shared decision making to support

treatment choices

Direct access to diagnostics for GPs

Why is it so hard?

A gap between theory and practice

“Improvement initiatives are sometimes planned on the hard high ground, but are put into effect in the swampy lowlands.”

- Marshall and others, 2016, BMJ Quality & Safety

© Kenneth Allen

Implementation needs to take wide

range of factors into account

• Context matters

• Requires rigorous framing of the problem and contextual factors that could influence feasibility and effectiveness

• Including influencing professional behaviour such as attitudes to risk

Source: Imison and others, 2012 25

Bed use

System governance factors

• Governance models

• Commissioner behaviour/ relationships

• Provider behaviour/ relationships

• Staff beliefs and values

• Leadership

Hospital factors (supply side)

• Access (rurality)

• Internal processes – admission, treatment and discharge

Community factors

• Primary care supply and capacity

• Community care supply and capacity

• Local authority care supply and capacity

Patient factors (demand side)

• Age

• Socioeconomic status

• Sex

• Health needs

• Beliefs and values

Community

An unequal battle?

Hospital

26

Primary and community care facing

significant challenges

27

• 1/3 GP practices have a vacancy for at least one partner

• 2016 - NHS England identified 20% GP practices as vulnerable

• 1/5 district nurse posts vacant

Savings are difficult to realise in

reality

28

• Some interventions identify unmet need = increase activity

• New services can fuel supply-induced demand

• Savings depend upon capacity being taken out of hospital

What should you be mindful of in implementing & evaluating an initiative?

When is evidence ‘good enough’?

30

• Hierarchy of evidence

• Pragmatism: rigour versus

reality

• What/who is the evidence for?

Source: NIHR Evidence Works

Top tips for implementing &

measuring change

31

1. Be realistic: Planning & implementing large-scale change takes time

2. Ensure aims are clear and be explicit about desired outcomes

3. Hospital use and cost are not the only measures

4. Be flexible: Monitor and measure as you progress and adapt accordingly

5. Understand your data: what do you have? What is it telling you?

Top tips for implementing &

measuring change

32

6. Relationships matter: sets of processes can only achieve change if underpinned by good relationships and a shared vision

7. Engage staff: ultimately, they will deliver the change

8. Invest in leadership

9. Don’t get bogged down in governance: let form follow function

10. Context matters and it changes

Conclusion

33

• This agenda presents a huge challenge

• Shifting the balance of care is not easy and will take time

• Nobody can argue against the principle of better, more appropriate care closer to home

• But we cannot assume that this will save money, especially in the short term

• To succeed, we need a relentless focus on what works & constant monitoring

www.nuffieldtrust.org.uk

Follow us on Twitter – twitter.com/NuffieldTrust

Sign up for our newsletter – www.nuffieldtrust.org.uk/newsletter-signup



Michael Lawrie

Operational Research and Evaluation Unit

NHS England


Lessons learned from evaluating

national transformation programmes

Michael Lawrie

Operational Research and Evaluation Unit, NHS England

www.england.nhs.uk

• NHS England’s Operational Research and Evaluation Unit was formed in 2015

• Around 35 mixed methods analysts working on a range of national programmes .

• Economists, social researchers, statisticians, operational researchers.

• We aim to embed robust evaluations, support the adoption of rapid cycle evaluation approaches (where suitable) and work alongside programmes to help them be as evidence based as possible.

• The new care models programme is currently our largest area of work.

37

Introduction

www.england.nhs.uk

• To provide some advice on how to undertake high quality evaluations, drawing on our experience of evaluating NHS England transformation programmes.

• The session will be structured around our five step rapid cycle evaluation approach.

• I propose to open up for questions after each of these five steps so that the session is as tailored to your queries as possible.

38

Aims and structure of the session


Rapid cycle evaluation Specify the desired outcomes and

the broad policy parameters within which the programme should be

implemented and, to the extent it exists, the evidence-informed

common recipe for change.

Describe how the programme is supposed to work. Articulate, through a logic model, the proposed

interventions and the causal chain linking them to

the desired outcomes.

Measure impact through comparison with what would have

occurred (the counterfactual)

Measure what changes are being made and what is happening as a

result.

Learn, adapt and improve


1. Specify the desired outcomes, policy parameters

and the evidence informed case for change

• Agree the evaluation questions early and prioritise them;

• Agree a range of outcome measures. Commonality will aid neatness when aggregating data (especially at a national level); but also embrace the variety with local sites defining local measures;

• Measure a few things well. But ensure that a varied mix of outputs and outcomes are established;

• Decide whether to focus evaluation resources in on a particular initiative, or on a full programme evaluation.


A set of simple evaluation questions

that we have tried to stick to…

What is the model and how is it intended to work?

What changes are being made and what is happening as a result?

What is the impact?

What is the cost?

What is causing the impact?

What should be replicated and spread?

How should x, y, z be implemented?


2. Describe how the programme is supposed to

work.

• We advocate the use of logic models to describe how a programme is supposed to work.

• It provides a hypothesis for the evaluator to test.

• Producing a logic model can help to tease out the full range of value of a programme.


The activity

funded connects

to…

The outcomes of

the funding…

And onto the

longer-term

impacts…

If we deliver training to people who are

unemployed then they will improve their skills;

If they improve their skills, then they will gain in

confidence or gain a qualification;

If they gain in confidence or gain a

qualification, then they will obtain more job

interviews and job offers;

If they get a job, then their income will rise, and

there will be reduced unemployment.


1. What is the rationale for your intervention?

[What is the problem you are trying to address?]

2. Given your rationale, what impacts are you ultimately trying to achieve?

[What would be the end measure of success for you?]

3. What short- to medium-term outcomes would generate those impacts?

[What immediate benefits would you like to see before you can achieve these longer-term success measures?]

4. What type of activities would generate those outcomes?

[What will you do differently / new practices will you introduce in your area to achieve these immediate changes?]

5. Given all of the above, what resources are required?

[What will you need to enable you to work differently / introduce new practices]


Inputs

These are the resources available to

deliver the intervention.

This may be in terms of specific cash

funding or ‘in-kind’ contributions.

It is relatively straightforward to put a monetary value on

inputs and, in a framework of costs and benefits, inputs

are the costs.

Evaluation at this level is about

economy and the resources consumed.

Activities & Outputs

These are the things that an intervention

does, e.g. provide workshops, practical support, training etc.

Outputs are quantitative

measures of this activity, e.g. No.s of:

• Beneficiaries•Materials distributed

• Sessions held

Evaluation at this level concerns

implementation and efficiency (the

relationships between inputs and outputs).

Short-termOutcomes

It is often useful to distinguish between short- and medium-

term outcomes.

Short-term outcomes can be defined as

changes in knowledge / awareness / attitude

– e.g. ‘beneficiaries have an increased awareness of ...’

This is based on a simple model of

behavioral change, which suggests that

these changes precede changes in

behaviour or condition.

Long-termImpacts

This is the final, high-level effect of the intervention – e.g.

‘Improved life expectancy, reduced health inequalities’.

This relates closely to the original rationale

for intervention.

Impacts are subject to a very wide range of

other contextual influences (e.g.

combinations of other policies, programmes, economic conditions),

- illustrated by the very permeable line

around this box.

Context to the InterventionThese are the wider economic, social, environmental, and policy conditions. This is very important: interventions do not

take place in a vacuum and contextual factors affect the intervention and its results.

Medium-termOutcomes

Medium-term outcomes are

changes in behaviour or condition – e.g.

‘beneficiaries increase levels of physical activity’

In describing any outcomes, language suggesting change

(‘increased, reduced’) is useful.

Evaluation here is about effectiveness.

The relationship between inputs and

outcomes is the basis for cost-effectiveness / cost-benefit studies.

Rationale for Intervention

This is the justification for the selected intervention, e.g.: what is the nature and scale of the specific problem being addressed? What will happen if we ‘do nothing’? Why this intervention and not alternatives?

Source: Midlands and Lancashire CSU

www.england.nhs.uk

• One of the central evaluation questions is to try to

understand what would have happened in the

absence of the programme intervention compared to

what did happen – the impact question.

• There are numerous ways in which to do this, and the

next few slides show varying ways in which to do this

based on real life examples.

46

3. Measure impact through comparison with what

would have occurred (the counterfactual)


Tracking the time series


Use of simple comparators


Use of statistical process control


Controlling to counteract regression

to mean

Source: Health Foundation


Looking at data at a more granular

level

Source: Midlands and Lancashire CSU


4. Measure what changes are being made

and what is happening as a result

• Outputs and early outcome measures are important

leading indicators of change. E.g. patient activation

E.g. Clinician training

Initial median score 51.00

Follow up median score 60.60

Median difference 9.60

Target: Average follow up score to be at

least 50 from April 2017

Source: Vanguard local metrics return

Source: Vanguard local metrics return


4. Measure what changes are being made

and what is happening as a result

• Input metrics are also important to consider – e.g.

what are staff doing differently in order to deliver the

changed service in question (skill mix, hours worked,

location of work).

• Qualitative insights are crucial here, both to

understand how things are changing and to explore

chains of causality. E.g. some of the vanguard

evaluations are using embedded evaluators

(Researchers in Residence).

www.england.nhs.uk

• Collate and synthesise all available evidence. Triangulate to draw firm conclusions about how the initiative (set out in the logic model) is working and test the causal chains therein.

• If resources are not available to undertake this robustly across a full programme, look in depth at particular parts of your programme – case studies.

• Set up feedback loops which will allow the evaluator / analyst to feed in regular findings to programme decision makers.

• Ensure a diversity of data are available – different stakeholders require different standards and types of evidence.

55

5. Learn, adapt and improve



Anne Jarrett & Fiona Russell


Local Partnerships & Local

Government Authorities


asdf

Fiona Russell

[email protected]; 07799 466328

Anne Jarrett

[email protected]; 07917 813829

25 April 2017

Supporting the evaluation of local

BCF schemes

mailto:[email protected]

mailto:[email protected]

BCF bespoke support programme for 2017-19

• Part of a suite of support commissioned by the national Better Care Support Team,

which also includes:

o Masterclasses and Regional Networking Events

o Regional Support Funding

Better Care Adviser and Multi-Disciplinary Support:

Programme objectives:

Further support:

• Delivered by LGA, as a national partner in the Better Care Support Team, with NHS

IMAS and Local Partnerships

• To secure agreement on a compliant BCF plan for 2017-19

• To support implementation of BCF schemes or performance improvements against

the BCF metrics

• To support local areas at all levels to drive their integration plans forward and achieve

greater integration of health and care, including preparing for graduation from the BCF

• To strengthen leadership to lead integration locally

The evaluation challenge… • Evaluation questions about local schemes are being framed in different

ways but there appears to be a real challenge in trying to answer them

Local BCF

schemes

Is the scheme

‘effective’?

Is it delivering

the outcomes

we expected?

How can we deliver more from

the scheme?

Is it contributing

to the achievement of BCF plan objectives?

Does it represent ‘value’ for

BCF investment?

Should we continue

the scheme?

The data challenge • Can we drawn on national data and best practice to develop a simple

framework to help support local areas with scheme evaluation?

• Can we drill down into the national data set to help frame questions about local scheme evaluation?

BCF NATIONAL DATA

• Can we identify and share effective approaches to local scheme evaluation?

LOCAL BEST PRACTICE

Develop a local framework • Evaluating the performance of local schemes

On a scale of 1 – 10 where 1 is “not at all” and 10 is “to a great

extent”, indicate the extent to which each scheme: N.B. Here are some example headings that we think should be included – however please adapt and add to these as you feel works for your local context.

Total up the

scores, and take

the lowest scorers

through the

process on the

next page.

Today’s exercise – developing a local framework

• Three tables – each taking on of the most common scheme types

(Reablement; Care at home services; Integrated care teams)

• The task is to populate a local framework, considering the following:

How would you know your chosen schemes are delivering:

• Improving outcomes

• Value for money in the long term

• Long-term capacity building for integration and new care models

• Improving patient/service user satisfaction

• Shifting provision towards prevention and community

What are the challenges of trying to answer these questions locally

What are you doing about evaluation that is proving useful to you?

What sort of support would you find valuable?



Susanna Shouls


NHS Elect

Measurement and analysis for

Improvement

Susanna Shouls Measurement & Analysis

www.nhselect.nhs.uk Twitter @NHSElect

http://www.nhselect.nhs.uk/




Why measure?

“The third habit is measurement and oversight. For many measurement of clinical operations is driven by external audiences: payers, regulators and rating agencies. Although high-value organizations share this reporting obligation, they primarily use measurement for internal process control and performance management.”

• More specific measures

• Customised to their processes

• Integral to accountability and strategic aims

Model for improvement

“Do something and see what happens” Paul Plsek, 2016

Mark Wallinger

Art on the Underground

Measurement throughout a project

Potential for improvement

Getting a baseline Did it make a

difference?

What needs to be in place to mainstream

improvement?

Evaluating worth

(benefits, costs)

A P

D S

A P

D S

A P

D S

A P

D S

A P

D S

Seven steps – measurement for improvement To reduce unplanned admissions to my acute acute hospital Outcome: ?number of unplanned admissions Process: ?number of patients seen in my alternative solution Balancing: ?a&e attendance ?satisfaction

Consistency is key here!

Baseline data …

Step 1: Decide your aim

Features of a good aims statement

A worthwhile topic

Outcome focused

Measurable

Specific population

Clear timelines

Succinct but clear

Adapted from

Tom Nolan in

The

Improvement

Guide

Beware Weasel words

“Weasel words are words that have no specific and obvious and singular meaning.

They bring no clear images to mind of what is meant.”

best practice, effective, evidence-based, excellence, high quality, high value, responsive, value, value-added, world class and many more!

Source: Stacey Barr quoting Don Watson

A word to watch out for

Does your aim include the word “by”?

For example:

“We want to reduce the number of complaints by giving our staff customer care training.”

Step 2: Select measures

?Already started?

?Use what exists?

Driver Diagrams Schematic view of a system on the left we depict outcome and as

we move right we drill down into the network of causes that drive the outcome, from ‘primary’ to ‘secondary’ drivers

Driver Diagrams

On the right we depict ideas for system changes that might ultimately impact the outcome. The diagram represents our theory about how to modify the system to change the outcome

Aim:

2 stones

lighter!

Energy Out

Energy In

Walk daily

commute

Stairs not

lift

Exercise

Reduce

alcohol

intake

Eat Less

Pedometer

Gym work

out 3 days

Squash

weekends

No pub

weekdays

Take

packed

lunch

Low fat

meals

Driver Diagrams - weight loss

Exercise 1: Discuss your aims and your key drivers

• You have 10 minutes

Understanding populations

2x2 matrix approach

Was the patient

suitable for AEC?

Was the patient seen in

AEC?

Yes

No – admitted

Yes

Right patient, right place

Missed opportunity

No

Wasted resource

- admission

- A&E

- ?avoid attendance

Right patient, right place

Assessing right place, right time of care.

Using AEC “same day emergency care” to illustrate the 2x2 matrix

Does this graph help us?

How about this?

Celebration of success???

• If 1/3 patients seen in AEC would not have previously been admitted ….

• But avoiding admission in 2/3 …

• Is this efficient or not? And how did we know?

Qualitative approach

Obvious learning points

Number of patients seen in AEC is not a sufficient to demonstrate impact – although is an easy / obvious process measure

Understanding impact on total population not just our service

Qualitative data helps to understand the “why” patient is in right place / “wrong place” as well as assessing effective processes

Steps 3-5 reflections

Improving quality in the English NHS

“Reflect, measure and learn rapidly about what is and is not working to help implementation become more successful”

“In complex adaptive systems like the NHS and its component health care organisations, learning is dynamic, plans need to be flexible and strategies emergent if they are to be effective”

Improving quality in the English NHS – A strategy for action

Kings Fund, Feb 2016

Don Berwick, Jennifer Dixon, Chris Ham

It is a waste of time collecting and analysing your data if you don't

take action on the results

Step 6 - Review Measures

How, when and where you review your measures is a key

challenge for you to take away from today

You may not get it right first time!

You may need several iterative attempts through steps 4, 5 and 6

And finally - Step 7

Resources

Guide to measurement for improvement

http://www.nhselect.nhs.uk/What-we-do/Service-Improvement--Measurement/Measurement-for-Improvement

Link to video and guide (free)

H l o io ’s v l io w o co si

http://www.health.org.uk/publication/evaluation-what-consider














Any questions?

BCF Scheme Measurement and Evaluation Workshop · PDF file 1 The Better Care Fund BCF Scheme...

Documents

Transcript of BCF Scheme Measurement and Evaluation Workshop · PDF file 1 The Better Care Fund BCF Scheme...