Update on National Social Care User Experience SurveySSRG Conference, 8-9 March 2010

Suzy Powell (DH), Juliette Malley (PSSRU), Paul Niblett (NHS IC)

The role of experience information

Results

Experience

Activity

Costs

Results+

Experience= Outcomes

Outcomes+

Activity

Costs

= Analysis of service contribution and value for money

‘Experience’ is a key element in the cluster approach to outcomes accountability. The most obvious way to find out what outcomes have been achieved for people is to ask them – and this requires survey programmes and perception-based PIs.

Ends

What did we do?How well did we do it?Inputs & outputs

Is anyone better off?Result of activity

What was it like for the user?Perceptions

How much did it cost?Budgets & funding

Mea

ns

The current surveys

What’s wrong with the old programme?

• No formal rolling programme – survey groups agreed in advance• No consistent repetition to track progress• Data limited to specific client group or service type• Variability in how representative data is of wider service• Outcomes focus is not consistent: not always asking the right questions• Not aligned with the NHS or other survey programmesSo…

Many groups’ experience not covered

Poor value performance information (both nationally for NI127 and NI128, and locally for monitoring own services)

No common basis for local partners

Little support for delivering the outcomes of Putting People First

The concept

Our work aims to design and develop new national surveys for adult social care users which better reflect the wider agenda:

Focusing on whole user populations;

Asking questions about experience of key outcomes, in line with policy priorities and including those which go beyond core social care services;

Repeated annually to track progress and act as a basis for planning and Local Area Agreements;

Providing a basis for national benchmarking, but flexibility to support additional local surveys;

Giving further opportunities to link to other NHS and community surveys through common outcomes.

Report on Development Work – Brief History

• Discussion paper – proposals for method

and content of survey PLUS key challenges

• Presentations and opportunity for feedback

• Response paper – proposed method and

development work

Proposed method

• Cross-sectional SRS all social care clients

• Self-completion questionnaire

• Address potential non response by

– maximising opportunities for sample members to get

help to complete questionnaire

– using a validated version for PWLD

• Questions on social care-related quality of life

(SCRQOL) + factors influencing

• MANY CHALLENGES + QUESTIONS

Development work

1) Implications of having ‘help’– Who helps and how, question revisions, additional questions to

help interpretation & guidance

2) Including residents of care homes– Factors influencing RR & appropriateness of q’aire, guidance &

revisions/additions to questions

3) Accessible self completion for PWLD– LD version & factors influencing RR, guidance

4) Can advocates be used to help people answer where they don’t have anyone else to help?

– Guidance

Study 1: Having help to complete

• Method– Cognitive interviews to test questions– Service user & helper interviewed together

• Key findings– Type of help given very varied e.g. hold pen, read

questions, translation into FL, proxy– Helper can influence responses given

• Dominance in relationship

– Type of help influenced by disability of user• Lack mental capacity – proxy response

– Questions about ‘feelings’ difficult for proxies• No obvious frame of reference• Produce random answers, inflate variance + SEs

Study 2: Care homes

• Method– Interviews with care home managers– (Cognitive) interviews with residents

• Key findings– All residents could express some opinions about their

care– Nearly all need some kind of help

• Writing, reading, understand meaning, formulate responses

– Survey format difficult for many• Fitting answers to response options• General terms difficult, need personalising

Study 2 continued

• Key findings (continued)– Staff most appropriate people to help

• Many residents lack regular visitors• Helper needs to know day-to-day life of resident well• Advocates patchy + expensive• Relatives/staff/advocates all could bias answers

– Gaining informed consent • Approach of care homes not consistent• Level of ‘informed’ consent required

– Variety of methods for dealing with mail– Some questions need modification for care home

residents e.g. “home”

Study 3: LD version

• Method– Focus groups with PWLD to design questions– Interviews with PWLD to test questions

• Key findings– Most PWLD cannot read

• at all/level required for main q’aire• use of illustrations & simplified language helped with

understanding

– Still many will need help to complete• Help to understand meaning & formulate responses

– Survey format difficult for many• Fitting answers to response options• General terms difficult, need personalising• Summing across different aspects also difficult

Study 4: Advocates

• Methods– Interviews with LA UES managers & advocacy agency

managers

• Key findings– Very little experience of using advocates in this way– Availability of advocacy patchy– Risks associated with home visits– Familiarity with clients needed, requires several visits– But many advocacy agencies enthusiastic about survey

and providing such support

Recommendations for pilot

• Encourage people to seek help to prevent high rates of non response– Someone who knows user well/see regularly– Care staff in care homes– Not advocates unless existing relationship

• Reduce potential for helper to bias results– Directions on front sheet & to care home staff– Include questions on who helped and how in q’aire– Consider survey for carers

• Users who lack capacity to consent should be excluded– Responses from proxies to ‘feelings’ questions (i.e. main measure)

are unreliable– Use care managers to identify who lacks capacity to consent

• Test and cost method for excluding

Recommendations continued

• Engage care home managers to prevent high rates NR– Send letter to CH manager– Let CH manager know which residents receive

survey– Encourage CH staff to help residents– Reassure not using results to assess CH

• Adapt questionnaire for CH residents

• Use LD version for PWLD

Recommendations for analysis

• Check sample not biased due to exclusion of people lacking capacity– Requires detailed population data– Stratified sampling may be better method for future

• Check for non response across different groups– Particularly from CHs, PWLD, people with high levels of need,

people living alone– High rates of NR may indicate method not applicable

• Check for unusual results or differences between group with and without help– Large differences may indicate a problem, require further

analysis/data collection– Alternative methods may be indicated

Recommendations for future work –dependent on pilot results

• Test & cost alternative methods for CH and PWLD– E.g. semi-structured interviews, observation

– Consider developing proxy questionnaire

• Test stratified sampling method

• Test & cost telephone helpline with trained operatives to conduct interview

• Consider developing proxy questionnaire

• Reconsider role of advocates as advocacy becomes more embedded

Pilot Survey

• Pilot survey – 3 options

• All service users

• Residential care

• Service users with LD

• Timescales

• Draft guidance and questionnaire available

• Awaiting ethical approval before final guidance issued

• Fieldwork in April

• Return data to NHS IC by 13 June 2010

Timetable• Report on pilot to SSUSG in late July

• Ethical review for full survey in early September

• Guidance and questionnaire issues in October

• Fieldwork in February 2011

• Data returned to NHS IC in late May 2011

• Provisional data used for NIs available for regulator and councils (via NASCIS) on a pre-release access basis in mid June

• Provisional data for NIs published in mid August 2011

• Provisional full survey data published end September 2011

• Full survey report published December 2011

• Other development work going on throughout this period to improve survey for 2011-12