DEPARTMENT OF HEALTH ERIC REVIEW OCTOBER 2014. Document structure Background Profile Overall...

DEPARTMENT OF HEALTHERIC REVIEW

OCTOBER 2014

2

Document structure

• Background• Profile• Overall Perceptions of the ERIC Dataset• Providers of Information• Users of Information• Changes and Improvements to the ERIC Dataset• Reports and Analysis Tools• The Way Forward

3

BACKGROUNDDH – ERIC Review – October 2014

4

Introduction

• The Estate & Facilities is critical to the delivery of high quality and safe clinical services by the NHS• It covers areas such as cleaning, food, environment quality, energy and utility usage• Annually £7.2 billion of revenue is spent on its estates & facilities by the NHS• Circa. £5 billion of capital is spent to maintain and improve it

• The Estates Related Information Collection (ERIC) is the sole source of data on the NHS Estates & Facilities asset base and patient and business critical services in England and is critical to many stakeholders supporting many functions including:• The NHS constitution pledge ‘The NHS commits to ensure that services are provided in a

clean and safe environment that is fit for purpose, based on national best practice’ through providing key data on the healthcare environments

• Informing Estates and Facilities policy development within the Department of Health• Contributing to the Governments ‘Efficiency, Reform and Growth’ Agenda by driving up

estates and facilities efficiency through the use of key performance indicators and peer benchmarking

5

Overview of Project Purpose

• The data collection covers 18 topic areas and comprises approximately 100 data fields and questions

• Data is collected at both organisation and individual site level where the site meet a given criteria

• Individual organisations can provide information for a number of different sites under their control

• The most recent data collection provided over 140,000 individual data items

• In addition to a light touch annual review a more fundamental review of ERIC is undertaken periodically to ensure data collection remains fit for purpose and meets the needs of the Department of Health, NHS bodies and other stakeholders

• The fundamental review has two key research elements that will define any changes to the ERIC data collection:

• Market research to collect and analyse the views of NHS users and other stakeholders

• Research undertaken by the Department to define their own internal requirements and those from other government departments

6

• Existing Dataset• What organisations use ERIC for and any benefits it brings to the running of the

organisation• The extent to which users are aware of the benefits of ERIC• The extent to which users are supportive of ERIC in supporting public accountability and

transparency• The extent to which data requested is readily and easily available• The level of confidence in data accuracy against topic areas• Are users aware of the analysis produced by DH and if so, do they find the analysis useful• The extent to which organisations use ERIC to benchmark and improve estates and facilities

performance• ERIC IT System

• Data entry and user friendly – Could the way data is entered be improved?• Reporting – Which reports are used most and why? Can their format be improved?

• Improvements• Any questions/topic areas users wish to be removed and why?• Any questions/topic areas users wish to be added and why?• Any existing questions that would benefit from being broken down into a more detailed

level of granularity and why?

Objectives

7

Methodology Summary

• Stakeholders from NHS bodies were invited to take part in an online survey• Stakeholders that completed the survey were given the option to take part in a follow up

telephone depth interview• A selection of those willing to be recontacted were called to discuss their views on ERIC in

more detail• Other stakeholders (e.g. Regulators, arm’s length bodies, professional bodies, etc) were also

invited to take part in a telephone depth interview

493 NHS body invites

33 Qualitative Interviews

178 Survey Completions

Other Stakeholders

8

Quantitative Methodology - Sampling

• OMB were provided with 504 individual contacts covering 255 different organisations– Contacts were either a DoE Contact or an ERIC contact

• Contacts were sent a pre contact email from the DoH project manager in order to introduce the research as well as giving them an opportunity to opt-out, update the survey contact to someone more appropriate, etc.

• Following the pre contact email exercise there were 493 contacts remaining to be sent the survey link

Total

Total contacts provided 504Unusable – pre contact email opt-outs 11Total contacts invited 493Unusable – incorrect email 9Total useable sample 484Interviews achieved 178Response rate 37%% Organisations covered 63%

9

Quantitative Methodology – Survey Progress

15-Sep

16-Sep

17-Sep

18-Sep

19-Sep

20-Sep

21-Sep

22-Sep

23-Sep

24-Sep

25-Sep

26-Sep

27-Sep

28-Sep

29-Sep0

5

10

15

20

25

30

35

40

21

35

84 3

36

18

11

3 4

35

Number of Interviews

• The online survey was open for two weeks from 15th September to 29th September• Responses peaked in the first 24 hours (15th/16th) and again following the two survey

reminder emails (22nd and 29th)

10

Qualitative Methodology Details

• Following completion of the online quantitative survey, 33 telephone depth interviews were conducted with stakeholders

• The sample was made up of• NHS Bodies that had taken part in the online

quantitative survey• Contacts were selected based on their

response to the online survey to achieve a balance of both providers/users and positive/negative views

• Other Stakeholders including regulators, arm’s length bodies, professional bodies, other governmental departments and commercial organisations

• The discussion guide was drafted alongside the DH project manager and aimed to explore stakeholder’s views on ERIC in greater depth in order to enable a more detailed understanding of the overall of experience of providers and users

11

PROFILEDH – ERIC Review – October 2014

12

Respondent Profile - Online Survey

The online survey covered a wide range of organisations, with the majority of respondents employed by Acute Foundation Trusts and Acute NHS Trusts

Base: All Respondents, Other (178, 2%)

Organisation Type

Acute Foundation Trust

Acute NHS Trust

Mental Health Foundation

Mental Health Trust

Ambulance Foundation Trust

Ambulance NHS Trust

Care Foundation Trust

Care Trust

Community Trust

Social Enterprise

42%

26%

12%

5%

1%

2%

1%

1%

7%

2%

13

None (0%); 56%

Some (1%-50%), 37%

Most (51%-99%); 4% All (100%); 1%

Sites under a PFI contract?

Base: All Respondents (178)


14

Respondents to the online survey were typically in an ‘Estates and/or Facilities’ role


Job Type

Manager of Estates/Facilities

Estates/Facilities Officer

Director of Estates and/or Facilities

Assistant Director of Estates and/or Facilities

Information Officer

Director of Finance or similar

Other

37%

20%

14%

12%

11%

2%

4%


15

0-2 years; 18%

3-4 years; 15%

5 or more years; 67%

Length of Relationship

A large proportion of the sample have been a provider/user of ERIC for 5 or more years.The majority of stakeholders surveyed classified themselves as both a provider and a user.


Provider; 22%

User; 2%

Both; 75%

Provider or User?


16

Respondent Profile - Qualitative Interviews

Organisation type No. of interviews

NHS Trusts

Acute NHS Trusts 7

Acute Foundation Trusts 4

Mental Health Foundation Trusts 7

Mental Health Trusts 1

Ambulance Foundation Trusts 1

Ambulance NHS Trusts 1

Care Foundation Trusts -

Care Trusts -

Community Trusts 2

Social Enterprise 1

Other stakeholders

Arm’s Length NHS Bodies 4

Professional Bodies 1

Commercial 4

Total 33

Among NHS Trusts, mix of:• Users and providers of ERIC

data (most were both)• Broadly supportive vs. less

supportive of ERIC (based on survey responses)

• Trust with and without PFI contracts

17

OVERALL PERCEPTIONS OF THE ERIC DATASET

DH – ERIC Review – October 2014

18Perceptions of the ERIC dataset depend on a range of inter-related factors

Perceptions of the ERIC dataset

• These factors all contribute to the trade-off between burden of collection and usefulness of data

• They also impact one another:• E.g. lack of awareness of potential

use/value can drive disengagement and less reliable data collection

• Increasing support for ERIC and improving overall impressions needs to be holistic• Dealing with both practical issues with

the collection process and with awareness/understanding of the data’s potential

19The Big Picture: Support for ERIC is strong, but it is not considered perfect

• General support for the idea of ERIC– Delivers accountability– Useful for policy makers and for

Trusts– It is the best we have and has

the potential to be very useful• Usability of the system and burden

of providing data has improved in recent years

BUT

• Problems with the realities of delivery degrade trust in the data and likelihood to use

I am fully supportive of ERIC

I am somewhat supportive of ERIC

I am somewhat unsupportive of ERIC

I am very unsupportive of ERIC

43%

41%

9%

2%

Thinking about your opinion of ERIC overall, which of the following statements best applies?

Base, Don’t know: All Providers/Users (176, 5%)

To drive greater support, usage and impact, improvements are required to ‘tighten up’ the quality and consistency of data and to increase depth of understanding of how it is used and why it is important.

Rationalisation of the dataset would help, but only limited agreement on areas to cut back.

20

Overall perceptions of ERIC

“It provides information that is key to decision making both centrally and

locally. It’s an important starting point to gain insight into our own and

other organisations.”

“We have insufficient confidence that the data is accurate, making

benchmarking and comparisons of doubtful value”

“It can be like a big black cloud hanging over you from April to June

where you know you have to get things done, but if you plan you can

take the cloud away.”

“We certainly need to keep it. It is the best we have got and has got a lot

better. We need to work on improving it even more.”

21

It allows for better benchmarking

Provides us with organisation data/behaviour

Do not trust data interpretation/data quality

Useful - general

Strategic planning

Useful for FOI requests

Helps inform government policy

Other

46%

37%

20%

13%

7%

3%

3%

9%

Highlights the importance of data quality and the fact that support for ERIC is somewhat qualified

Base: All supportive of ERIC (147)

Thinking about why you are supportive of ERIC, please could you provide us with more details?

(Top mentions - 3%+)

Reasons for support of ERIC highlight some key strengths but also reveal concerns about data quality

Spontaneous mention of poor data quality even when asked for reasons for support

22

Do not trust data interpretation/data quality

Outdated/needs updating

Time consuming/onerous

Isn't relevant to our organisation

Other

40%

20%

20%

10%

15%

Further evidence that data quality is a key concern

Base: All not supportive of ERIC (20)

Thinking about why you are not supportive of ERIC, please could you provide us with more details?

Concerns regarding data quality are the primary reason for lack of support

23

Perceptions vary by organisation type

Based on the qualitative interviews, certain types of organisation within the NHS benefit less from ERIC and are more aware of problems and difficulties completing the return…

Community Trusts

Ambulance Trusts

Social Enterprises

• ERIC not set up to record data in a way which suits their Estates– Multiple smaller building– Varied and dynamic usage within sites

• The burden of data collection/collation is greater

• The reliability and ‘meaning’ of the data is limited

• Benchmarking is not useful, as not comparing like for like (vs. other types of Trust) and the pool of similar organisations in so small

24

PROVIDERS OF INFORMATIONDH – ERIC Review – October 2014

25

How Trusts manage the process

Receiving and checking new

data definitions

Collecting and collating data

Checking and chasing data

Inputting and submitting the

return

• Single Estates (usually) professional responsible

• Most share new definitions with contributors

• But draft definitions can confuse, so may wait for final version

• Some attending workshops (but not all)

• Emailing contributors (other departments and external providers) in advance and then sharing definitions

• Some re-work spreadsheet to increase clarity for some staff

• Mining and re-working own databases (some collecting data on a monthly basis)

• Responsible individual required to chase (usually by email) some departments or external organisations

• Data checked before inputting against own internal systems and in some cases historic data

• Inputting usually the sole responsibility of one individual

• Undertaken sporadically over a few weeks

• Sign off comes from Estates or Finance Director

26Overview of perceptions of the process are mixed, but majority are satisfied

Things have improved, but concerns remain and Trusts generally agree that some improvements should be made

x Potential to misinterpret remains a core problem

x Time taken to receive final definitions delays process

x Data not always reliable/trustedx Takes time to receive all data (esp. from

3rd parties)x Staff providing data not always aware of

the real purpose/importance of ERIC

x Validation could be smoother and quicker

x Data entry still quite long-winded (additional fixes could smooth the process)

Many have been tightened up in recent years

Receiving draft definitions in advance useful to some

Number of fields reduced, so easier and quicker than it used to be

Staff become used to what is required and some Trusts have processes in place for streamlining collection

New interface an improvement (faster scrolling, fewer pop-ups)

27

Data collection satisfaction

Please could you rate your satisfaction with the following aspects of the data collection process...

Ease of use of the ERIC system

Managing the process of the submission

Overall Experience

Data collection at Site level

Data entry, validation and submission

Data collection for aggregate sites

Interpretation of the data definitions

13%

9%

9%

6%

9%

5%

4%

61%

63%

62%

56%

50%

42%

37%

17%

20%

22%

25%

24%

20%

35%

6%

8%

6%

10%

15%

14%

21%

2%

1%

1%

2%

2%

3%

3%

0.006

0.01

0.146

Very satisfied Satisfied Neither Dissatisfied Very dissatisfied Not relevant

Base: All Providers (172)

% Satisfied

74%

72%

71%

62%

59%

47%

41%

A high proportion are satisfied with the ease of use, submission process and overall experience of the data collection process. However, only two fifths are satisfied with the interpretation of data definitions.

28Moving forward: Summary of improvements and changes to the submission process overall

• Tighten data definitions• Speed up delivery of final definitions• Make definitions spreadsheet easily

adaptable (sharing with others)• More integrated calculations• Ability to deactivate irrelevant fields• Ability to annotate return with caveats

and issues• Enable validation ‘as you go’• Carry data forward/view last year’s values• Import from Excel

• Reinstate forums• Reinstate/refresh/publicise roadshows

and seminars• Produce introductory literature/manual• Raise awareness of ERIC’s importance

and potential benefit within Trusts– Encourage adoption of business

processes which are integrated/aligned with ERIC data requirements

…relating to dataset and process…

…relating to provision of support and information…

29

Difficulties in completing the return

Have you had any issues with the following when completing your ERIC return...?

Obtaining information from within your organisation

Data not available in time for submission

Time required to manipulate data to align with ERIC

Time taken to validate data

Time required to interpret data definitions

Obtaining information from Private Finance partner/outside organisations

Obtaining information from NHS Property Services

Anything else?

70%

51%

49%

45%

41%

30%

27%

8%

7 in 10 stakeholders have had issues in obtaining information from within their organisation.5 in 10 have struggled to get data in time for submission.

Base: All Providers (172)

30

Improvements to submission process

Would you like to see any improvements made to the overall submission process?Would like to see improvements to sub-

mission process?44%

34%

22%

Yes No Don't know

Of those giving a ‘valid’ response, the majority would like to see improvements made to the submission process.A large proportion (22%) of ‘don’t know’ responses suggests a lack of clear understanding of what needs to change but an

underlying sense that the system isn’t perfect.Base: All Providers (172)

“The data definitions are quite complex and could

do with being a little simpler in explanation”

“The input system could be easier, there’s a lot of stepping through screens

and waiting for validation”

“There remains too much guesswork – there is a lack

of standardisation in interpreting the data definitions between

organisations”

31

Make definitions clearer/more detailed

Improve software/navigation/make more user friendly

Add tools/resources for collecting data e.g. Notes, a manual, a forum

Smarter validation/Improve validation

Ability to blank/deactivate fields which aren't relevant

Ensure consistency/accuracy of data input

Ability to report on individual sites

Reduce required information required for submission

Carry data forward where it is unlikely to change

Break down sections to make collection simpler

Provider training in how to complete ERIC

Other

29%

20%

11%

9%

7%

7%

4%

4%

4%

4%

4%

20%

The most commonly suggested improvement to the submission process is improving the clarity/detail of data definitions

Base: All who would like to see improvement (75)

Improvements to submission process

Would you like to see any improvements made to the overall submission process?(Top mentions - 4%+)

32The Challenges of the Process: Trusts report a number of challenges associated with ‘getting the ERIC return in’

Concerns• New staff= inconsistency in data and in

slowing down of the process• Junior staff not engaged in ERIC and

do not see it a priority• Dealing with multiple sites and landlords is

very time-consuming (esp. Community Trusts and Ambulance Trusts)• Dealing with NHS Property Services

considered particularly difficult this year

• Final definitions arrive late, delaying start of process

Potential solutions/improvements• Address issues with ambiguity around

NHS Property Services’ role and remit

• Provision of ERIC introduction literature or training (bring back or better promote roadshows and seminars) to drive understanding and engagement to staff in other departments

• Speed up delivery of final definitions

“We get some draft definitions early on, but it seems to take a long

time to authorise them for use.”

33Data Definitions: The most commonly mentioned problem area

Concerns• Often thought to be the root of

inconsistencies and inaccuracies in the data as a whole

• Often definitions leave too much room for interpretation (e.g. large cost measures not stipulating what is/is not included)

• Trusts choose what suits their existing systems and approaches, and what suits their strategic aims

• Definitions can be long and complicated• Unclear which external providers/internal

department’s data to include• Complex, key metrics are tricky to calculate

Potential solutions/improvements• Greater clarity in some key definitions

• Provide ‘include…; don’t include…’ examples

• Take care to maintain brevity• Provide integrated systems to

calculate complex metrics• Reinstate online forums / FAQs section

to help with interpretation issues• Make definitions spreadsheet easier to

adapt for end users

“There used to be online forums to help with the definitions, but they stopped them. They were helpful, and we need

them back.”

34

Data definitions

“Take transport miles. Which miles belong to whom? A&E blue light are

ours, but patient transport services…are they ours or the commissioning body’s

Are these things getting double counted?”

“As the years have gone by it has got easier.”

“There is just too much ambiguity. They need to spell out what should and what should not be included.”

35Data collection: General concerns are raised about collecting data

General Concerns• Takes time to receive all data back• Collecting from 3rd parties can be difficult

• E.g. PFI partners, contractors may not collect all data in format required

• Finance data not always collected at a site level or in required format

• Providing aggregated data is counter-intuitive for sites with multiple activities in different buildings

• Trust’s internal organisation of sites not matching ERICs (e.g. number of sites)

• In shared buildings data may be duplicated or missed

• System does not reflect building/site usage

Potential solutions/improvements• Some types of organisation should be

excluded from providing certain types of data (needs to be clearly set out in definitions and in the online system)

• Ability to annotate return and include caveats about data which may not be complete (e.g. when provided by external contractors etc.)

• Training and instruction to encourage Trusts to build requirements for particular data into contracts with suppliers

“ERIC is useless for us in a Community setting as it does not

take into account usage.”

36Data collection: Specific parts of the dataset are cause for concern

Quality of Buildings/Backlog Maintenance• Complex calculations required• Up-to-date surveys not always

completed for all buildings

Transport• Business miles not recorded

consistently• What/who to include not always clear

Waste• Contractors not collecting data on

individual sites/buildings

Finance• Ambiguity about what to include (e.g.

capital costs)• Ambiguity in terms of which staff to

include in hard/soft FM costs

Food Services• Staff vs. visitor catering costs hard to

define in canteens• Are sub-contracted staff canteens an

income or cost?

Mor

e co

mm

only

men

tione

d

Contracted Out Services• Ambiguity in terms of which staff to

include in hard/soft FM costs

37Limited agreement regarding the most problematic data areas

Greatest confidence in reliability is placed against Fire Safety, Trust Profile and Age Profile.There is low confidence in the reliability of data for Transport Services and Contracted Out Services.

Trust Profile

Contracted Out Services

Finance

Staff

Transport Services

Fire Safety

Cleanliness

Laundry and Linen

Areas

Function and Space

Age Profile

Quality of Buildings

CHP EnergyWater Services

Waste

Car Parking

Food Services

0%

10%

20%

30%

40%

50%

60%

70%

0% 10% 20% 30% 40% 50% 60% 70%

% V

ery

Confi

dent

inA

ccur

acy

% Very Easy to Provide

Ease of collection vs. Perceived Reliability (% Very Easy/Reliable)

• Few areas are considered very easy to collect

• Trusts are more confident about the accuracy of their data, but often concerned about data which they rely on third parties to deliver (esp. NHS PS and some PFI partners)

• Other broad areas of consistent concern are transport services, backlog maintenance and some aspects of finance

• Some admit making guestimates for some of this data in the past (and to an extent now too)

38

Ease of Data Collection

How easy is it to collate the data for each section of the ERIC return?Fire Safety

Age Profile

Trust Profile

Areas

Car Parking

Water Services

Energy

CHP

Cleanliness

Laundry and Linen

Staff

Function and Space

Waste

Food Services


Transport Services

Finance


26%

23%

30%

20%

12%

14%

13%

19%

10%

9%

9%

10%

8%

8%

10%

3%

6%

6%

49%

44%

33%

34%

40%

36%

33%

26%

35%

35%

34%

27%

28%

27%

24%

24%

20%

18%

20%

26%

30%

33%

36%

33%

33%

13%

38%

34%

42%

38%

30%

35%

40%

33%

45%

41%

2%

4%

3%

11%

7%

14%

17%

3%

10%

13%

9%

19%

26%

20%

18%

26%

22%

26%

1%

2%

1%

2%

3%

1%

2%

1%

3%

3%

3%

4%

5%

3%

6%

7%

4%

4%

1%

1%

2%

1%

1%

1%

1%

36%

2%

3%

1%

1%

1%

2%

1%

5%

1%

2%

Very easy Easy Average Difficult Very Difficult Not relevantBase: All Providers (172)

Stakeholders find collating the information for Fire Safety, Age Profile and Trust Profile easiest whereas information on Transport Services, Finance and Contracted Out Services present more of a challenge.

% Very Easy/Easy75%

67%

63%

54%

52%

50%

46%

45%

45%

44%

43%

37%

35%

35%

34%

27%

25%

24%

39

Accuracy of data collection

How confident are you in the accuracy of data you provide within ERIC?

Trust Profile

Fire Safety

Age Profile

Staff

Areas

Car Parking

Finance

Water Services

Cleanliness

Energy

Laundry and Linen

Waste



Function and Space

Food Services

Transport Services

CHP

52%

61%

51%

42%

41%

40%

26%

40%

36%

39%

31%

27%

23%

19%

24%

24%

16%

40%

45%

34%

44%

51%

51%

50%

63%

48%

52%

48%

55%

53%

55%

59%

53%

51%

58%

22%

1%

3%

3%

4%

7%

8%

8%

8%

8%

8%

9%

16%

17%

17%

18%

16%

18%

3%

2%

1%

1%

1%

1%

1%

1%

1%

2%

1%

3%

1%

1%

3%

1%

5%

4%

34%

0.012

0.006

0.006

0.023

0.029

0.006

0.017

0.023

0.029

0.023

0.023

Very confident Quite confident Column1 Not at all confident Not relevant Base: All Providers (172)

% Very Confident/Quite Confident

97%

95%

94%

93%

91%

90%

89%

88%

88%

87%

86%

80%

78%

78%

77%

75%

73%

61%

40


Thinking about the areas where you are less confident in the accuracy of data provided, why is this...?

We rely on others for some of the information

Accuracy of data is hard to verify/hard to get

Information required is too detailed/hard to quantify/interpret

Data isn't collected in a format suitable for ERIC

Collecting data is resouce heavy/time consuming

Historical data is provided to us

Colleagues don't understand the importance

Age of site/buildings

Question definitions changing

Information asked for isn't relevant to us

Other

45%

29%

18%

15%

11%

8%

3%

3%

2%

2%

6%

Stakeholders are least confident in the data they provide when they rely on others for this information

Base: All not confident in accuracy for at least one area (89)

41


Thinking about the areas where you are less confident in the accuracy of data provided, why is this...?

“Gaining access to the information across the trust for certain

information can be difficult due to a number of reasons like staff turnover,

etc.”

“Departments outside of my control i.e. Finance, Contracting, HR, etc. do

not see the relevance of the information and the importance of its

accuracy.”

“There is difficulty in getting reliable or even any data from building

owners where we are occupiers/tenants”

“As many of these services are outsourced to third parties, we are not able to verify the quality of the

information provided by them”

42

Data entry and and validation

Concerns• Validation:

• Time consuming to correct red flags (awkward to view data on single screen and only get alert at end of process)

• Not always ‘true’ errors (esp. for small Trusts)

• Cannot see last year’s data (for sense check)• Need to key in by hand – can’t upload from

other data sets• Insufficient built-in calculations to deliver

speed and efficiency (e.g. CHP metrics)• Need to enter values into irrelevant fields

Potential solutions/improvements• Enable validation checks ‘as you go’ to

speed up the process• Variable red flag tolerance for different

type/size of Trust• Annotate fields for self or colleagues to

refer to next year• Ability to view last year’s data alongside

this year’s or pre-filled field for static data (building sizes etc.)

• Enable importing of data from Excel• More auto calculations to produce complex

metrics• Use of a database which is completed an an

ongoing basis, generating ERIC data

43

USERS OF INFORMATIONDH – ERIC Review – October 2014

44

Usage of ERIC data: Overall picture

• Usage varies across Trusts, but most see some benefit

• Depends on personal preferences and level of engagement, as well as level of suitability of data for Trust types

• ERIC data provides value and has an impact on decision-making in many cases

• But reliability of data a concern for most and the primary barrier for not using more (or not relying on ERIC)

• General awareness of the potential uses of ERIC quite high but does not always mean a deep level of understanding of capabilities

• Some scope to reduce size of dataset, but lack of agreement on specific measures to cut

Increasing confidence in ERIC and raising awareness of capabilities and usefulness through education/communication should be a priority. Changes to the dataset itself likely to be relatively moderate

45

Moderate Users

Trusts describe different levels of overall use of ERIC data

Overall, Trusts are deriving some value from ERIC, and some are finding the data very valuable. There is scope to increase this through communication around its potential uses and improvements to reliability

• Using ERIC data quite regularly, for a number of purposes• But limited awareness and understanding of the range of

potential uses• ERIC not well known throughout the organisation

Enthusiastic Users

Avoiders/non-users

• Reliant on ERIC for strategic planning and other decision-making

• New applications and uses often sought• Known throughout the organisation (requests for supporting

info from ERIC quite common)

Mor

e co

mm

on in

sam

ple

• Doubts about benefits/suitability and reliability limit use• Seen as purely a data gathering exercise for DH• Most commonly smaller Trusts and Community Trusts

46

Benchmarking/Performance Management

ERIC data is used in the following ways within Trusts (1)

• Keeping track of performance against Trust KPIs and trends • Comparing key cost measures/performance against peer group• Identifying best practice in other Trusts

Reporting

Strategies and planning

• Inputting into annual reports to the Trust Board on KPIs or specific issues (energy consumption, carbon footprint)

• Informing the development of the Estates strategy (contributing to evidence for decision-making, along with other data)

Mor

e co

mm

on in

sam

ple

Enquiries & FOI requests

• Media enquires (most recently car parking and catering)• FOI requests linked to high profile topics or often by prospective

suppliers looking for a sales angle

Benchmarking and reporting on performance are considerably more prevalent uses. Trusts often use what ERIC makes easily available to contribute to their understanding and evidence base

47

Business Cases

ERIC data is used in the following ways within Trusts (2)

Using ERIC to drive operational decision-making is less common. Internal data often considered more appropriate and reliable

• To support plans for capital investment (assessing condition of existing estate to support decisions to refurbish or replace sites/buildings)

Contracts

Operational Policies &

Procedures

• Cost monitoring to inform procurement decision making (waste, energy, catering)

• Charge setting for other users of Estate (energy costs etc.)

• Benchmarking and change over time on metrics informing and supporting decisions such as energy efficiency measures or parking policy

Less

com

mon

in s

ampl

e

48

Use of ERIC

Have you used ERIC data in your organisation for any of the following...?Benchmarking/Performance Man-

agement

Reporting

Strategies and Planning

Public/Media Enquiries or FOI Requests

Business Cases

Contracts

Operational Polices and Procedures

Anything else?

91%

87%

69%

69%

59%

28%

28%

5%

Base: All Users (137)

49

Business case appraisal

Usage by other organisations: Arm’s Length and Professional Bodies*

• Contributing to data to evaluate business cases for (mostly) capital expenditure

*Note: Categories for purposes of anonymity

Understanding carbon reduction

performance

Monitoring and reporting performance on other sustainability

metrics

• Exploring, monitoring and reporting on carbon footprint for NHS at regional and national levels

• Encouraging and helping Trusts meet their carbon reduction goals and report on their performance in annual reports

• Water use and waste (recycling)• Travel plans and adaptions

Providing Estates advice to Trusts

• Benchmarking performance on particular areas to advise on operational decisions

50

Preparing commercially

available reports

Usage by other organisations: Commercial organisations

• Analysing the raw ERIC data and using to create relevant and usable benchmarking reports

• Trend analysis• Add commentary and descriptions which help Trust

identify any problems with unreliable data etc.• Preparing general Estates report on wider NHS

Services to specific Trusts

• Brokering deals for capital investment (new buildings, change of use)

• Specific projects to help Trusts deal with risk analysis• Helping with strategic planning and cost-cutting• Imbedding in Trusts in temp roles (Estates Directors, Fire

Officers, external auditor roles)• (also helping complete the ERIC return)

51

High impact / influence

Impact and benefits of using ERIC data: Impact on decision-making and strategic planning

Demonstrates that ERIC can have some impact and has the potential to be highly valued

Mor

e co

mm

on

Contributory impact / influence

No impact or influence

• Most common• Supports decision making (e.g.

contract negotiations) and inputs into strategic planning

• Other data also used in conjunction with ERIC

• ERIC data can kick start investigations into performance and expenditure

• Relying on ERIC data for some key decisions and strategic planning (high level within Estates dpt.)

• Used as basis for strategic business planning

• Moderately common

• May access ERIC data, but not said to have had impact on decisions or strategy

• Inferior to alternative data (internally collected)

• Lack of confidence in data• Least common

Increasing impact and benefit

52Impact and benefits of using ERIC data: Other benefits

These more ‘peripheral’ benefits could be promoted more as part of a wider communications strategy

• Need to collect data and provide reports (or similar) keeps Estates teams in touch with the wider Trust teams

• Encourages understanding and cooperation– Formally via benchmarking clubs or

informally through local contacts

Promoting best practiceDriving communication with other departments

and Trusts

• Benchmarking can flag up examples of top performing Trusts, triggering exploration of practices/site visits and senior level dialogue

• Requirement to complete return can encourage timely and accurate data collection

“It is useful to know what is going on with the Trust. I like to know how we

are doing and how we compare.”

“We have developed processes and a system to collect the data on an

on-going basis, that is just the right thing to do.”

53Impacts and benefits felt by the wider NHS and other stakeholders

Overall, other stakeholders support ERIC as it is established and unique. However, they agree that the system has the potential to be of even greater benefit to themselves and Trusts

• Provides unique breadth of data• A long historic dataset enabling useful trend

analysis• Although other sources of data exist, they do not

have the same breadth/length• Covers high level metrics of relevance to policy

makers

Enables departments to meet their statutory

requirements

Enables large scale, high level analysis across the

whole NHS Estate

Speeds up processes

Helps verify business cases

• Centralised, pre-collected data removes the need for targeted investigation and research when assessing business cases/strategies

• Provides a layer of assurance which otherwise might be missing from assessments

54Barriers to (and problems encountered when) using ERIC data

Awareness of what it can do

Lack of time to investigate/use

Concerns about reliability and accuracy

Other systems in place which are more suitable

Timing of data delivery too slow

Insufficient/unsuitable data to meet needs

Driving up awareness of ERIC’s capabilities should increase overall engagement and help contribute to increasing reliability and accuracy of data (which should also be addressed directly)

Wider System-centric

Two types of barrier

work together

and influence

one another

55Accuracy, reliability and completeness of data remains a challenge

Increasing reliability and accuracy is the key priority for the majority of stakeholders. There is acknowledgement that this is not a straightforward task

• While Trusts are fairly confident in the accuracy of their own returns (with some important exceptions), they often doubt the reliability and accuracy of the rest of the dataset

• Other stakeholders (NHS central departments, professional bodies and commercial users) are also somewhat sceptical, feeling that inaccuracies exist

• However, participants across the sample (especially other stakeholders) pointed out that at a macro level the data was mostly acceptable in terms of accuracy– Some added that the quality of data has improved over

time

“I think there are doubts about the accuracy right

across the data.”

“It is such a big dataset, you know that for high level

measures the mean figures are accurate.”

56Lack of accuracy in the data occurs in different ways…

While some high level changes to how ERIC is planned and delivered are mentioned, the need to retain continuity is also acknowledged. Ironing out data input issues is often seen as priority no.1

Inconsistencies Inaccuracies/omissions Systemic problems

Data input issues Inherent issues

• Difficulty interpreting data definitions

• Different internal approaches to data monitoring

• Difficulty collating correct data

• Trusts manipulating data to ‘look good’ or support strategic plans

• Lack of care collating, checking and inputting

• Not using audited/verified data

• Closed dataset = inability to correct inaccuracies in past data

• A snap-shot, so not up-to-date

• Lacks granularity• Missing chunks of data (NHS

PS)• Underpinned by a lack of engagement and understanding of

ERIC by those providing data• Exacerbated by churn in Estates teams and fragmentation of

NHS structure

57Improving ERIC for those using it is primarily about improving the data itself and raising awareness of benefits and importance

Some users also suggest that Trusts should be more accountable for the accuracy and completeness of their returns

• Roadshows, local meetings, forums• Training for new staff (and non-Estates staff)• Pro-active publication of case studies showing high

performance• A re-launch of ERIC to refresh perceptions• Tap into existing networks and organisations…gain

commitment of HeFMA, NHSPS, CHP, P21 to drive message to others

• Publicise (and act on) this review

Increasing accuracy

Driving up awareness of benefits/uses

Encouraging greater

engagement

58Specific improvements to the dataset from a user perspective

• Faster delivery of data• Consider how to deliver more frequent data collection, or ideally real time data

collection• Aligning more closely with…

– Other data collection systems and procedures– Govt/legal requirements around reporting on issues such as sustainable

development• Capture change in activity and usage data• Update definitions around renewables in line with latest recommendations from DECC

and Environment Agency• Users (those not involved in preparing return) would ideally like more data if possible

– Not always able to pinpoint what this is in interviews, but would welcome more granularity

59

CHANGES AND IMPROVEMENTS TO THE DATASET

DH – ERIC Review – October 2014

60Which parts of the ERIC dataset are most/least useful?

Suggest consideration given to cutting back some specific questions in conjunction with other internal evidence

• There is agreement from within all sample groups that ERIC could benefit from further ‘pruning’ to make it less complex and reduce the burden on Trusts

• But, there is no clear agreement on areas of data to remove on a large scale

• Trusts and other stakeholders are using a wide range of data for different purposes

• Removing some of the less useful and less reliable specific questions across the dataset is the recommended approach

“Those question on mileage are just not accurate and

not useful to anyone.”

• Transport the most consistently mentioned area suitable for removal of questions

• Contracted Out Services also more likely to be mentioned as unnecessary

61Suggested changes to data by section – Trust level data

Trust Profile• No consistent suggestions recorded, but

overall usefulness to Estates often questioned

Contracted Out Services• Increase granularity (break-out by

contractor type)• Or remove altogetherFinance• No consistent suggestions recorded

Staff• Break down staff costs further within both

soft and hard FM costs

Transport Services• Remove business mileage (T06 03) and

patient mileage (T06 04)• Remove number of claims through HTCS

(T06 02)Fire Safety• Section considered a duplication of other

returns…is this needed?

Cleanliness• Split out pay/non-pay costs (T0801)• Greater links to clinical issues such as

infection rate/risks

Laundry and linen• Low area of interest in non-acute orgs

Other suggestions:• Does the Trust have a sustainable development action plan/how developed is it?• Capture incidence of overheating in wards• Capture use of nitrous oxide used (greenhouse gas)

62Suggested changes to data by section – Site level data

Area• Break down occupancy costs• Site heated volume (S0104) very

inaccurate, could be removed

Function and space• S0201/2 deliver unreliable

result…clarify/simplify or remove

Age Profile• No consistent suggestions recorded

Quality of Buildings• Risk backlog difficult to calculate, but

potentially useful data. Clearer definitions a priority

CHP• Should be filtered and only asked to

relevant Trusts

Water Services• Separate surface water and highway drainage

Waste• Reduce options for different types (S082-5)

Car Parking• Include more granularity on parking charging

structures• Collapse staff and patient parking as often not

separatedFood Services• Include relevant PLACE scores

Energy• Record who energy provided by/how rates

negotiated• Ability to deactivate irrelevant energy

consumed fields• Harmonise with CRC requirements• Include gas consumed locally

63

Usefulness of ERIC

How do you rate the usefulness of the information provided within each section of ERIC?Energy

Water Services

Waste

Areas


Cleanliness

Function and Space

Food Services

Finance

Car Parking

Laundry and Linen

Age Profile

Fire Safety

Staff

Trust Profile


Transport Services

CHP

50%

42%

39%

43%

42%

39%

39%

40%

39%

33%

30%

26%

26%

24%

18%

20%

14%

14%

42%

47%

48%

45%

44%

46%

46%

43%

43%

44%

43%

47%

46%

46%

52%

47%

39%

31%

4%

7%

9%

8%

10%

12%

12%

11%

14%

17%

20%

21%

21%

23%

20%

23%

36%

16%

2%

3%

2%

3%

3%

1%

2%

1%

2%

4%

4%

4%

4%

4%

7%

5%

8%

7%

1%

1%

1%

1%

1%

1%

1%

4%

2%

1%

2%

2%

2%

2%

2%

4%

2%

32%

Very useful Fairly useful Not particularly useful Not at all useful Not relevantBase: All Users (137)

64

Use of ERIC

Which section of ERIC do you use for each of the areas you listed? (Top 5 Sections)

Strategies and Planning

Base 75

Quality of Buildings 57%Function and Space 56%Areas 55%Age Profile 51%Trust Profile 41%

Benchm’ing/Perf. Mgmt

Base 109

Cleanliness 57%Waste 55%Laundry and Linen 54%Finance 50%Food Services 50%

Reporting

Base 96

Fire Safety 44%Energy 36%Water Services 34%Areas 32%Function and Space 31%

Business Cases

Base 43

Quality of Buildings 35%Finance 30%Age Profile 30%CHP 19%Function and Space 16%

Public Enq’s/FOI Requests

Base 42

Car Parking 40%Food Services 36%Staff 24%Energy 19%Trust Profile 17%

Contracts

Base 18

Contr’d Out Serv’ 39%Laundry and Linen 22%Finance 17%Cleanliness 17%Energy 17%

Operational Pol’s/Proc’s

Base 75

Water Services 38%Fire Safety 25%Waste 25%Car Parking 25%Staff 13%

65

REPORTS AND ANALYSIS TOOLSDH – ERIC Review – October 2014

66

Awareness of reporting and analysis tools

Trusts would benefit from further information and guidance regarding the potential uses and differences between different types of reports and analysis on offer

• Trusts are generally aware of most benchmarking reports (and the majority use most or all)

• However, Trusts are not always fully aware of what the reports offer, or their potential benefits

• Awareness of KPI analysis is slightly lower, and lack of understanding/ confusion with benchmark reports is evidentAw

aren

ess

Understanding

Benchmarking Reports

KPI Analysis

“I know I work with ERIC all the time, but I’m not really sure what the differences are.”

67

Awareness of reports and analysis tools

Which of the following benchmarking reports are you aware of...?Site Comparison

Trust Key Performance Indicators

Organisation Wide

Region Trusts

Region Wide

Organisation Trend

93%

92%

87%

84%

82%

81%


Estates & Facilties KPI's 66%

Are you aware of the ‘Estates and Facilities KPI’ analysis produced by DH?

68

Usage of Benchmarking Reports and KPI analysis

• Vast majority of Trusts use at least one benchmarking report annually

• Most run the reports once per year, soon after the publication of the data (often included in annual reports)

• Some also run reports more frequently to meet specific operational requests within the Estates or Finance departments

• KPI analysis less frequently described, and sometimes the difference between benchmarking reports and KPI analysis not understood

“We always use ERIC KPI’s if challenged with anything to

prove our efficiency. If it does not answer my needs I look

elsewhere.”

Smaller Trusts and Community trusts describe

less interest in using reports

69

Usage of Benchmarking Reports

How frequently do you use these reports...?

The majority of stakeholders use the benchmarking reports no more than 2-3 times a year

Site Comparison

Trust Key Performance Indicators

Organisation Wide

Region Trusts

Region Wide

Organisation Trend

0.01

0.01

0.01

3%

6%

4%

2%

1%

1%

14%

17%

14%

13%

12%

13%

36%

37%

33%

36%

34%

34%

35%

36%

36%

34%

36%

37%

10%

3%

10%

12%

13%

14%Weekly Monthly 4-6 times a year 2-3 times a yearOnce a year Never

Base: All Aware

70Perceptions of using ERIC data to benchmark are generally positive but with some variations

• Useful means of making high level comparisons

• Degree of flexibility in what to benchmark against (but can result in comparing apples and pears)

• Ability to export data in spreadsheet format delivers flexibility (for those able to manipulate it)

What is working? What is not working?

• Insufficient detail/flexibility:– Unable to compare just specific metrics – Unable to select specific Trusts to compare

against– Unable to generate bespoke reports

• Radar charts unintuitive/difficult to interpret• Benchmarking only a snapshot in time• Content and purpose of reports not always

obvious until you have read the contents

“We want to be able to select specific Trusts, not just ones that the system suggests, and not just

locally.”

“It is particularly useful as it shows how we perform

nationally but also specifically within the local area.”

71

Benchmarking between organisations

How useful do you think the current ERIC system is for benchmarking performance against comparable organisations for each of the topic areas ERIC covers?

Energy

Water Services

Cleanliness

Waste

Food Services

Laundry and Linen


Areas

Finance

Fire Safety

Function and Space

Car Parking

Staff

Trust Profile

Age Profile


Transport Services

CHP

36%

31%

31%

30%

30%

26%

23%

29%

31%

20%

22%

24%

18%

19%

14%

20%

13%

9%

50%

53%

53%

53%

53%

54%

55%

47%

45%

54%

47%

44%

50%

47%

50%

42%

47%

37%

11%

12%

13%

13%

12%

15%

16%

18%

20%

20%

24%

26%

27%

23%

26%

31%

28%

22%

3%

2%

2%

3%

3%

4%

5%

4%

3%

5%

5%

5%

3%

9%

9%

5%

10%

17%

Very useful Fairly useful Not particularly useful Not at all useful Base: All Users (137)

The Energy, Water Services, Cleanliness, Waste and Food Services sections are most useful for benchmarking. Contracted Out Services and Transport are less likely to be seen as useful for benchmarking, presumably linked to their perceived

reliability

72Estates and Facilities KPI Analysis is seen as moving in the right direction

Links to clinical data such as infection rates would make KPI analysis more useful

“It is going the right way, but I think that ERIC could look to what is going on in the commercial world and

learn something.”

• An interesting output• Delivery via dashboard monitoring helpful• But KPIs not linked to clinical data and not always intuitive/relevant

73A number of specific improvements to the access and delivery of ERIC data are suggested


• Benchmarking to include a toolkit to interrogate the data– More detailed benchmarking bandings (e.g. ability to just focus on FM costs)– Ability to choose specific Trusts to compare against

• Clearer titling and description of report contents and potential benefits• Ability to benchmark against other, non-NHS organisations • Multifunctional rather than broad brush reporting (via a tool kit for users)• Template reports which could be applied to different sections and modified• Access to the ERIC online system/functionality for wider NHS departments• Greater access to more information via guest log-in (e.g. all potential FOI requests)• More commentary and signaling of implications within benchmark reports (more of a ‘so

what’ element)• Adherence to latest open data principles

74

Access improvements

Would you like to see any improvements in how information from ERIC can be accessed?

Whilst the proportion suggesting improvements is relatively low (12%) a significantly high proportion (31%) say ‘don’t know’ suggesting a lack engagement and clear understanding of what needs to change

“Any of the information that would be available on a freedom of information request should be available via a guest

login.”

Improvements to Access of ERIC Info' Required? 12% 57% 31%

Yes No Don't know


“I would like to be able to print off comparisons from neighbouring

Trusts.”

“More intuitive user reports. Currently you have to open most reports before understanding what they are telling you in order to find the right one to

suit your needs”

75

Benchmarking improvements

Are there any ways in which you think the benchmarking potential of ERIC could be improved?

“I think the information within ERIC should identify where top 20%

performance is being achieved and seek a narrative/case study from the

providers to share best practice.”

“Training should be given as to exactly what is required in the data fields.

The written guidance is often confusing and difficult to understand.”

“The data granularity or the way it is presented does not lend itself to

useful benchmarking. I feel that with the right metrics, it could be slicker

and much more useful.”

“All the data collected by ERIC is very useful, but the difficulty comes to

comparing apples with apples. Too many decisions are made when

comparing apples with oranges.”

Base: All Users(137)

76

THE WAY FORWARDDH – ERIC Review – October 2014

77

The Way Forward (1)

• Support for, and satisfaction with ERIC are reasonably high• But improvements to the data definitions and collection process along with increased

engagement, understanding and awareness of the system in general/what it is capable of are required

• Increasing accuracy and consistency in the data will be a challenge and requires both improvements to the definition and support/education/training for those tasked with collecting the information

• Some further improvements to the ERIC IT systems (inputting and reporting) will be helpful, but recent improvements are already noticed

• A slimmed-down, less complex dataset is desirable in general. But, given recent cuts in the size of ERIC, identifying clear areas to make large scale cuts is challenging– Transport Services and Contracted Out Services present the most obvious opportunity

for cuts• Some additional detail to the data is also required to make ERIC more usable• At a broad level, there is agreement that ERIC could deliver greater value if it were more

clearly integrated with other data collection and reporting systems, and focused more on providing a holistic picture for Trusts (making greater linkages to clinical outcomes)

78

The Way Forward (2)

• The future in terms of coverage of Community Trusts should be considered carefully– While stakeholders recognise that a consistent, fully comprehensive dataset is desirable,

important questions are raised regarding the fit between ERIC data definitions and the way in which these Trusts operate

• It will also be important to clarify the situation regarding NHS PS going forward to ensure Trusts know what to do and that users of data can be confident that the dataset is representative

• While awareness of many aspects of ERIC (reporting options, usage by government departments) is numerically high, further work is required to increase depths of understanding and engagement with ERIC overall– This will deliver greater reliability of data, as the importance of completing the return

consistently and accurately will be better appreciated– And will drive up usage of, and advocacy for ERIC as a source of useful data

DEPARTMENT OF HEALTH ERIC REVIEW OCTOBER 2014. Document structure Background Profile Overall...

Documents

Transcript of DEPARTMENT OF HEALTH ERIC REVIEW OCTOBER 2014. Document structure Background Profile Overall...