Patient experience survey – adult primary care:

Methodology and procedures

February 2017

© Health Quality & Safety Commission 2017

Health Quality & Safety Commission PO Box 25496, Wellington, New Zealand

This document is available on the Health Quality & Safety Commission’s website:

www.hqsc.govt.nz

Contents

1. Introduction .................................................................................................................... 6

1.1 Background ................................................................................................................. 6

1.2 Purpose of the survey .................................................................................................. 7

1.3 Patient experience survey project ................................................................................ 7

1.3.1 Pilot phase ....................................................................................................... 7

1.3.2 Project governance .......................................................................................... 8

1.3.3 PHO service agreement process ..................................................................... 8

1.3.4 Phased rollout .................................................................................................. 8

1.3.5 Evaluation ........................................................................................................ 9

1.4 Participation ................................................................................................................. 9

2. The survey tool ............................................................................................................ 10

2.1 Survey development .................................................................................................. 10

2.2 Domains of patient experience .................................................................................. 11

2.2.1 Question classification and scoring methodology ................................................ 11

2.2.2 Domain performance rating ................................................................................. 11

2.2.3 Coordination sub-domains .................................................................................. 12

Overall GP and nurse rating ......................................................................................... 12

2.2.3 Score calculation ................................................................................................. 13

2.3 Patient contact and demographic information ............................................................ 13

2.3.1 Long-term solution – NES .............................................................................. 14

2.4 Patient- and practice-identifiable information ............................................................. 14

2.4.1 Data access ................................................................................................... 15

2.5 Informing patients of the use of their information ....................................................... 16

2.6 Official Information Act requests ................................................................................ 17

2.7 National survey and report system............................................................................. 17

2.7.1 National system flexibility .................................................................................... 18

3. Survey process ............................................................................................................ 19

3.1 Process chart – interim .............................................................................................. 19

3.2 Process chart – final (NES) solution .......................................................................... 20

3.3 Data flows and system access ................................................................................... 21

3.3.1 Interim solution .............................................................................................. 21

3.3.2 NES long-term solution .................................................................................. 22

3.3.3 In situ solution ................................................................................................ 23

....................................................................................... Error! Bookmark not defined.

3.4 Collection method ...................................................................................................... 26

3.4.1 Pilot analysis ....................................................................................................... 26

3.5 Sampling method ....................................................................................................... 27

3.5.1 Email notification and online collection ................................................................ 27

3.5.2 SMS notification and online collection ................................................................. 27

3.5.3 In situ collection .................................................................................................. 27

3.6 Frequency of collection .............................................................................................. 28

3.7 Eligible patients ......................................................................................................... 28

3.8 Sampling ................................................................................................................... 29

3.8.1 Sample size ........................................................................................................ 29

3.8.2 Reminders........................................................................................................... 29

3.8.3 Sampling method ................................................................................................ 29

3.8.4 Demographic weighting ....................................................................................... 30

3.8.5 Non-response bias .............................................................................................. 32

3.9 National system provider services ............................................................................. 33

4. Use of the survey results ................................................................................................. 35

4.1 National accountability ............................................................................................... 35

4.1.1 Calculation of system level measures ................................................................. 35

4.2 Online reports ............................................................................................................ 36

4.2.1 Accessing the survey results ............................................................................... 36

4.2.2 The dashboard .................................................................................................... 36

4.2.3 Opening the reports ............................................................................................ 36

4.2.3 Viewing reports ................................................................................................... 37

4.2.4 Printing reports .................................................................................................... 38

4.3 Quantitative and qualitative analysis .......................................................................... 38

4.4 Continuous improvement ........................................................................................... 38

5. Annual review ................................................................................................................. 40

Appendix 1: Primary care patient experience survey ........................................................... 41

Appendix 2: Domain questions and picker scoring .............................................................. 56

Coordination ................................................................................................................ 56

Physical and Emotional Needs ..................................................................................... 58

Communication ............................................................................................................ 60

Partnership .................................................................................................................. 62

Appendix 3: Patient experience – interim solution – patient data extract ............................. 63

Data extract rules ............................................................................................................ 63

File transfer ..................................................................................................................... 64

Data fields ....................................................................................................................... 65

DHB codes ...................................................................................................................... 66

Appendix 4: Correspondence with patients ......................................................................... 67

Email correspondence ..................................................................................................... 67

Reminder email ............................................................................................................... 68

SMS correspondence ...................................................................................................... 69

Survey introduction .......................................................................................................... 69

Survey conclusion ........................................................................................................... 70

Appendix 5: Bias in response .............................................................................................. 71

Appendix 6: Survey data file................................................................................................ 76

Appendix 7: Patient experience report examples ................................................................ 88

Appendix 8: Licensed software support ............................................................................... 94

1. Introduction

1.1 Background Patient experience is a vital but complex area. Growing evidence tells us patient experience is a good indicator of the quality of health services. Better experience, developing partnerships with consumers, and patient and family/whānau-centred care are linked to improved health, clinical, financial, service and satisfaction outcomes.1234

With this in mind, the Health Quality & Safety Commission (the Commission) would like to make patient experience part of our measurement of health care quality and safety. Patient experience is a component of our quality and safety indicators framework.

Until August 2014, there was no consistent national approach to collection, measurement and use of patient experience information on a regular basis. The Commission designed a 20-item adult inpatient survey, which began in August 2014, as part of addressing this gap. Patient experience measures are now routinely in place for hospitals, providing both quantitative and qualitative results. The survey runs at least quarterly in all district health boards (DHBs) and covers four key domains of patient experience: communication, partnership, coordination, and physical and emotional needs.

In December 2014, the Ministry of Health (the Ministry) and the Commission began work to introduce patient experience measures for primary care using online patient surveys. The System Level Measures concept began with the Integrated Performance and Incentive Framework (IPIF). The refresh of the Health Strategy provided an opportunity to extend and evolve the IPIF concept to the whole of health system.

The Ministry worked closely with the sector to co-develop the new System Level Measures that provide a system-wide view of performance. The new measures engage the health sector more broadly (professions, settings and health conditions). IPIF is therefore transitioning to the ‘System Level Measures Framework’ to reflect this broadening of approach and the increased focus on value and high performance.

Patient experience of care is one of the four new System Level Measures to be implemented from 1 July 2016. For 2016/17, uptake of the primary care patient experience survey can be used as one of the contributory measures.

1 Picker Institute Policy Position no. 3: Why patients should share in decision-making? Oxford: Picker Institute Europe. URL: http://www.pickereurope.org/Filestore/Policy/position_papers/Picker_Policy_3_Why_patients_should_be_involved.pdf 2 Balik B, Conway J, Zipperer L, et al. 2011. Achieving an Exceptional Patient and Family Experience of Inpatient Hospital Care. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement. URL: www.IHI.org 3 CAHPS® Consumer Assessment of Healthcare Providers and Systems. 2010. The Clinical Case for Improving Patient Experience. URL: https://www.cahps.ahrq.gov/Quality-Improvement/Improvement-Guide/Why-Improve/Improving-Patient-Experience.aspx 4 CAHPS® Consumer Assessment of Healthcare Providers and Systems. 2010. The Business Case for Improving Patient Experience. URL: https://www.cahps.ahrq.gov/Quality-Improvement/Improvement-Guide/Why-Improve/Business-Case.aspx

1.2 Purpose of the survey The primary care patient experience survey, or PES, is being developed by the Commission to find out what patients’ experience in primary care is like and how their overall care is managed between their general practice, diagnostic services, specialists, and or hospital staff.

The survey looks at a patient’s experience of the whole health care system using primary care as a window. It focuses on the coordination and integration of care, rather than just the last visit to a GP’s surgery.

Being able to capture, understand and act on patient experiences in a timely manner is a vital contributor to improving health service delivery and also in prioritising attention and resources. The online survey and real-time reporting enables patients to have a voice and health teams that care for them can hear it through a direct and timely link.

The PES is a useful tool for practices, primary health organisations (PHOs) and DHBs to identify what is being done well, as well as areas for improvement. The survey results contain both quantitative and qualitative information. The comments from respondents provide a wealth of information that is not included in the scoring.

1.3 Patient experience survey project The PES project involves the following work to develop, test, and implement the survey:

1. Survey tool development, including cognitive testing and cultural appropriateness. 2. Sampling method. 3. Patient contact and demographic information. 4. Privacy impact assessments – interim and final. 5. PHO service agreement process. 6. Survey and reporting system. 7. Communication and engagement with stakeholders. 8. Establishment of a governance structure. 9. Project and survey evaluation.

While the project is a long way through its life span and the survey has begun, it is still in the implementation phase. The survey is yet to be rolled out across all PHOs and implemented via the National Enrolment Service (NES) system as the long-term solution to source patient’s contact information for the survey.

1.3.1 Pilot phase

Six PHOs agreed to work with the Commission through the pilot phase: Procare Networks, National Hauora Coalition, Midland Health Network (cognitive testing process only), Whanganui Regional Health Network, Compass Health and Pegasus Health.

Sixteen practices participated in the pilot, which consisted of eight survey rounds conducted between July 2015 and October 2015. Two of the Whanganui practices conducted in situ surveys during this time. In situ surveying, where patients can complete the online survey on a tablet while at the practice, is being piloted to see if this method captures the experience of patients that email or text surveying is unlikely to reach.

With the exception of in situ surveying, the pilot phase is now complete and has been evaluated. The evaluation of the pilot survey can be found on our website.

1.3.2 Project governance

Originally governance for the PES sat with the IPIF’s Joint Project Steering Group. With this group moving into abeyance in mid-2015, there was a governance gap. Ultimately governance of the PES should link to the governance structures for System Level Measures, but while this is being properly established, an interim solution was put in place.

The interim PES governance group acts as the decision-making body for the implementation of PES across PHOs and practices, and over the information that the survey collects. The group comprised people appointed by the PHO Services Agreement Amendment Protocol Group (PSAAP) parties, and covering the following skill sets/constituencies:

• PHO chief executives • Commission senior manager • general practitioner(s) • Ministry of Health Deputy Director-General • DHB senior manager • Royal New Zealand College of General Practitioners representative • consumer representative.

1.3.3 PHO service agreement process

PHOs ‘successful use and implementation of the patient experience tools, once those tools are developed by the Health Quality and Safety Commission’ was agreed to be added to part G IPIF of the PHO services agreement in 2014. The PES will be adopted by all practices as part of the PHO services agreement. However, there will be a phased rollout.

During the course of the PES project, the PSAAP has been briefed and the project team has worked with the Ministry and General Practice New Zealand (GPNZ) to identify where changes to the agreement are required to facilitate the survey process. The interim PES governance group reports to PSAAP and the Ministry.

At this point in time, aside from reference in part G, only changes to the referenced document ‘PHO enrolment requirements’ have been recommended to accommodate the collection of enrolled patients contact details (email and cell phone contact), or preference to opt out, for PES purposes.

1.3.4 Phased rollout

The first online PES, using the NES as a partial data source, began in November 2015. Initially only the five pilot PHOs participated in the survey using the ‘interim’ process. The interim solution required each PHO to provide a data file containing contact details for eligible patients, through a secure transfer, to the survey system provider (Cemplicity).

As NES is implemented and patient contact information is captured in the NES system, enrolled patients from all PHOs will be invited to participate in the survey. Patients will have given their explicit permission for their contact information to be used for the survey purposes when it is captured in the NES at the practice and the Ministry will provide a single file to Cemplicity.

The survey process is described in detail in the survey process chart below.

1.3.5 Evaluation

Both the survey and project need to be evaluated at appropriate points in time. The following process and timing will be followed for this:

• Pilot evaluation completed by 31 March 2016 and published on the Commission website

once reviewed. • Evaluation of the survey tool and reporting system. This is to be reviewed after a year’s

worth of surveying (four survey cycles). An independent party will be contracted in early 2017 to complete the evaluation with all stakeholders. The scope of the evaluation will be agreed with the PES governance group and is likely to include:

o user feedback via a survey developed by the independent party, facilitated by Cemplicity

o interviewing a selection of key stakeholders o the independent party analysing the results o a sub-group possibly being formed to recommend changes; this may be done by

workshop o question changes possibly requiring cognitive testing with patients (a longer

process). o the PES governance group approving any changes to be made.

The timeframe for completing changes will depend on whether cognitive testing is required or not.

Ideally we would have the majority of PHOs and DHBs using the survey before it is reviewed, however this may not be possible. The review process will need to be timely.

1.4 Participation The PES will be adopted by all practices as part of the PHO services agreement. However, there will be a phased rollout beginning with practices that are members of the five pilot PHOs. This is due to the method of collecting patient contact details via NES currently not being available. The five pilot PHOs can determine their own internal process to roll out the national survey across their contracted network of practices. A key constraint is the PHOs’ ability to easily access the required patient contact data in practice management systems, as permission needs to be sought from general practices to extract the information.

Once the NES is in widespread use and practices are populating their patient preferences field, more PHOs will be able to participate in the survey. NES will supply the patient contact details, replacing the need for a PHO to send data directly to Cemplicity. For details see section 2.3.1.

A patient will not be sent a survey until their practice and PHO have informed Cemplicity they wish to participate in the PES.

2. The survey tool

2.1 Survey development The primary care survey builds on the design, system and processes developed for the adult inpatient survey. As the survey looks at a patient’s experience of the whole health care system using primary care as a ‘window’, the survey is tailored for the ‘out of hospital’ environment. It focuses on the coordination and integration of care, rather than just the last visit to a GP’s surgery.

The survey is modular, with questions about medicines, diagnostics, specialists (other than GPs), allied health and support care, emergency department, hospital care and chronic conditions. Patients only answer questions relevant to their experiences. For example, questions on medication and chronic conditions will be answered only by patients for whom this is relevant.

The survey has been under development since 2014, with multiple iterations and review steps. In 2014, the Commission partnered with the Australian National Health Performance Authority for the survey development. The initial survey received from Australia in March 2015 underwent a preliminary feedback process with the pilot PHOs, the Ministry and the Royal New Zealand College of General Practitioners. It was revised to reflect the New Zealand health care environment. This work built on a number of tested and used tools that were evaluated by an expert advisory group. While each country is now developing its own survey, we are continuing to collaborate with Australia and keep track of survey alignment.

Patient feedback is voluntary and anonymous.

Point Research was commissioned to cognitively test, refine and evaluate the draft survey tool. The testing occurred between April and June 2015. Cognitive testing was used to understand how patients understand and interpret questions and instructions. The aim was to find out from patients if:

• the survey instructions are easy to understand

• the questions are easy to understand • the questions are relevant • the survey enables them to talk about what

they think is important • any important questions are missing.

The assessment included cultural appropriateness and tests for suitability in New Zealand’s primary health care context. Patient views on survey length, use of personal email or mobile contact information and completing the survey online were captured.

The survey underwent three phases of cognitive testing with patients from the enrolled populations. The survey was refined between each round, and refinement of the tool

between those rounds. The first phase involved cognitively testing the questions with patients in 15 focus groups from the six PHOs, covering the following population groups:

• Adults aged 25–64 years (2) • Māori (2) • Pacific peoples (2) • Asian (1) • Refugee new migrant (1) • Older adults aged 65+ years (2) • Younger adults aged 18–24 years (2) • Women (2) • People with disabilities (1).

The second phase tested the draft survey questions online with a wider set of patients from PHOs and health professionals. The third phase tested the revised survey with patients in individual interviews.

The survey changed significantly as a result of this process and the final outcome is the survey in Appendix 1. The rigor of this stage of survey development significantly informed question wording and designed the structure of the survey to be both clear to the consumer and informative for the provider. An online ‘dummy’ version of the survey as it appears to patients can be viewed and completed here http://bit.ly/1ONCOgt.

2.2 Domains of patient experience The Commission’s approach uses four domains (coordination, partnership, physical and emotional needs, and communication) to provide a consistent structure to measure patient experience in different care settings. These four domains communicate that a high-quality experience for patients depends upon high-quality and effective communication, a real partnership, excellent coordination of care and meeting both physical and emotional needs.

2.2.1 Question classification and scoring methodology

The four domains are comprised of various survey questions. Coordination consists of 14 questions; physical and emotional needs,15; communication, 12; and partnership, 6. All answers are assigned a value based on the Picker scoring methodology5 (eg, 10 = Excellent, 0 = Poor; 10 = Yes/Completely, 5 = Yes/To some extent, 0 = No). A complete list of these questions and their answer values is shown in Appendix 2.

2.2.2 Domain performance rating

The national average performance ratings for each domain are displayed at the top of the online reporting dashboard. The national average performance ratings are broken into Māori national average and non-Māori national average to enable a quick overview of any equity gaps. Practice, PHO or DHB domain results are displayed beneath this, depending on which user is logged on.

5 http://www.cqc.org.uk/sites/default/files/20151125_nhspatientsurveys_scoring_methodology.pdf

2.2.3 Coordination sub-domains

Effective coordination of care means the patient experiences the care pathway and the way information is conveyed as seamless. As coordination is an integral part to a patient’s experience of care, there are two sub-domains within this category. Excellent coordination reduces barriers to care and improves continuity of care.

Within the survey, there are four questions that relate to barriers to care and 10 questions that relate to continuity of care. The sub-domain scores are calculated in the same way as the domain scores.

Overall GP and nurse rating

The national average shown in the overall GP/nurse rating is calculated by averaging all patient responses to the statement ‘Overall, was your experience with your GP or nurse clinic’ – rank from very poor (0) to excellent (10).

Within the reporting portal, users have the option of applying filters, eg, your PHO, against the national average.

2.2.3 Score calculation

The question scores are calculated by adding the ‘score calculation’ of all responses and dividing by the total ‘number of responses’. Below are examples of how the performance ratings are calculated for each question (Table 1 and Table 2).6

Table 1: Calculating question score example – domain: coordination

Survey question: Has cost stopped you from seeing a specialist doctor?

Response option

Number of responses

Percentage of respondents

Score assigned

Score calculation

No 350 87.5 % 10 3500 Yes 50 12.5 % 0 0 TOTAL 400 8.75

(3500/400)

Table 2: Calculating question score example – domain: coordination

Survey question: Were you involved as much as you wanted to be in decisions about the best medication for you?

Response option

Number of responses

Percentage of respondents

Score assigned

Score calculation

Yes, definitely 300 75% 10 3000 Yes, to some extent

75 18.75% 5 375

No 25 6.25% 0 0 TOTAL 400 8.44

(3375/400)

At the domain level, the aggregation works in the same way.

Table 3: Calculating domain score example – domain: coordination

Survey question: Has cost stopped you from seeing a specialist doctor? Response option

Number of responses

Percentage of respondents

Score assigned

Score calculation

No 350 87.5 % 10 3500 Yes 50 12.5 % 0 0 Survey question: Were you involved as much as you wanted to be in decisions about the best medication for you? Yes, definitely 300 75% 10 3000 Yes, to some extent

75 18.75% 5 375

No 25 6.25% 0 0 TOTAL 800 8.6 (6875/8

00)

2.3 Patient contact and demographic information

6 Both tables have been adapted from: Care Quality Commission. 2015. NHS Patient Survey Programme: Survey scoring method. Retrieved from http://www.cqc.org.uk/sites/default/files/20151125_nhspatientsurveys_scoring_methodology.pdf.

In an effort to minimise costs and administrative burden for PHOs and practices, patient data will be obtained through the NES. The data collected includes patient contact information and demographic information such as age, gender and ethnicity. Only information that is directly needed for the purpose of this survey is received. The national system for obtaining patient information from practices and PHOs is described in Appendix 3.

2.3.1 Long-term solution – NES

The NES will be used to provide patient information from a single source. When practices update their practice management systems software for NES, it contains a ‘patient preferences’ field. This will enable practice staff to ask patients for their email and mobile contact information and whether they wish to opt out of the national survey. It is envisioned that practices will populate the patient preferences field during survey weeks, gradually building up their number of enrolled patients with saved preferences. Patient preferences only need to be captured for patients aged 15 years and over who have a consultation during each survey sample week (see timetable).

The method for obtaining patient contact details used in the interim solution by the pilot PHOs automatically opts all eligible patients into the survey each quarter. It has been agreed that pilot PHOs will continue to use this method until the patient preferences field is changed to populate automatically. Auto-population of patient preference fields from existing mobile and email contacts in practice management systems will reduce administration time and is likely to increase survey response rates. Options for auto-population are currently being explored.

2.4 Patient- and practice-identifiable information An interim privacy impact report has been prepared for the programme and reviewed by the Office of the Privacy Commissioner. Once the survey is using patient contact data from the NES, the final privacy impact report will be prepared, reviewed and published.

All responses to the survey are voluntary and anonymous unless responders choose to provide their contact details because they wish to talk to someone at their general practice. All notices and correspondence relating to the survey make this clear.

Each survey has a unique identification which enables line-by-line analysis of responses. When the patient data extract (whether from a PHO or NES) is imported to the national system, a number is assigned to each line of information. Neither the national survey nor the reporting process requires patient-identifiable information to be held in the database. Patient contact information is needed only initially to allow email and text correspondence to be addressed individually. Once each survey is closed, all identifiable information is deleted from the system. Demographic information is retained only to enable a comparison from time to time of who is not responding to the survey. Cemplicity is required to host the database within New Zealand and strict privacy and security protocols are maintained. Routine system penetration tests are run to maintain security.

The reporting system uses filters to allow PHOs, practices and DHBs to tailor reports. The Commission has incorporated features to mitigate the risk of identification through these filters, such as using age bands and not showing data or comments where there are fewer than five responses.

During the pilot surveys the Commission noted responders have a tendency, more so than during the adult inpatient survey, to mention the name of their practice/doctor/nurse. There is an option to review and moderate patient comments and remove names prior to reporting. A process for reviewing patient comments has been agreed with the PES governance group and is outlined in this document.

2.4.1 Data access

Data access rules decided by the PES governance group are in place. Different system users have access to different levels of information. The data access matrix is shown below.

Table 4: Data access matrix

What can be seen Who can see it Information Data level Patient Practice PHO DHB National Public Patient data file (ex PHO/NES)

Individual – identifiable

Survey responses

Individual – identifiable only if patient requests contact & approves access to survey

Online reports Practice – their own, identifiable

Practice – others in their PHO, anonymously

All PHOs, identifiable All DHBs, identifiable Online patient comments

Individual (anon) – practice level, their own

Individual (anon) – PHO level, their own

Individual (anon) – DHB level, their own

Individual (anon) – national level

Survey data file Anonymous responses Published reports

High level, national aggregate information

Table 5: Identifiable data access matrix

Organisation Role Can view Cannot view

Practice • General manager • Practice manager • General practitioner • Nurse • Administration team

• Their own practice’s results and comments.

• Other practices’ results within their PHO (unidentifiable).

• Results by other PHOs.

• Results grouped by DHB (in a way similar to national users).

PHO • Quality manager/lead • Clinical director • Primary care

manager

• All practices within their PHO. • Patient comments by practice

(identifiable). • Results by all PHOs.

• Results by all DHBs.

DHB • Planning and funding • Quality and risk

managers • DHB alliance

representative

• Results for practices and PHOs in their area.

• Results for the PHOs for whom they are the Lead DHB (eg ADHB can only see Procare).

• Comments for practices in their area (unidentifiable).

• Results for all PHOs. • Results for all DHBs.

National • Health Quality & Safety Commission

• Ministry of Health

• Filter by lead DHB and DHB of domicile.

• Patient comments by PHO and DHB (identifiable).

• Patient comments by practice.

The quarterly survey data files will be limited in accordance with the standard report access. At a national level, the Commission receives this data file for three purposes:

1. Calculating the weightings and the weighted reports. 2. Long-term analysis of trends. 3. Placing a qualitative lens on the survey results as done for the adult inpatient publication

here.

There are currently no published reports as the survey is currently implemented by only a small number of practices and PHOs. The governance group will agree the content of these reports and when it is appropriate to begin publishing.

2.5 Informing patients of the use of their information PHOs and general practices are well aware of the Health Information Privacy Code 1994 (HIPC) and the need to inform patients of the use of the information they provide. When patients enrol with a practice they sign an enrolment form agreeing to the enrolment process and are informed how their information will be used.

Given this process will have occurred some time ago for many patients, the Commission needs people to be informed specifically about the survey. During the pilot phase the Commission specifically tested a range of ways to achieve this, although it is mindful no single method will ensure all patients are fully informed. The privacy impact assessment discusses this in detail. A summary follows.

• Each quarter, participating PHOs and practices are provided a ‘Getting started’ pack by the Commission. The pack reminds them of key dates and provides the following process information:

o consent to extract data from the practice (applicable for interim solution) o frontline staff guidance (one page) o a display poster o a survey slide that can be added to a TV slideshow if applicable for the practice o a flyer to be handed out to all eligible patients during the survey sample week.

Practices can choose to text patients seen during the sample week to remind them they may receive a survey invitation. This is optional due to cost.

• Practices are encouraged to ask for patients’ email addresses during the sample week (individual rather than family). Emailed survey invitations contain more information for the patients, are referable at a more convenient time, and have a significantly higher response rate than SMS requests.

• Practices can use a classification code to record in their practice management system if a patient advises they do not want to receive a survey, eg, NOPES. This applies to the interim solution only.

Once a practice has adopted the NES software, their practice management system will be updated to allow practice staff to ask patients for their survey contact information and whether they wish to opt out of the survey. The Office of the Privacy Commissioner has advised that this proposed process is well inside the privacy rules: patients are notified, give their consent to receive the survey and have the ability to say ‘no’. All emails sent by Cemplicity have a clear ‘unsubscribe’ option and if a patient clicks the unsubscribe button, no further emails will be sent to that email address.

Once the survey is more commonly used, the Commission will publish national and local media releases to inform patients and encourage participation. The Commission also has information publicly available on its website here.

2.6 Official Information Act requests If Cemplicity receives an Official Information Act (OIA) request, it is required under its agreement to refer the request to the Commission.

If the Commission receives an OIA request, the governance group will be informed. If multiple PHOs’ data is sought, the governance group will be referred to for decision. Where an individual PHO’s data is sought, the PHO concerned will be referred to for decision.

2.7 National survey and report system Cemplicity has been contracted by the Commission to provide the national survey and reporting system that supports the survey. The ‘system’ describes the process to import patient information, send survey invitations, receive and store patient responses (anonymously), and provide real-time reporting to authorised people.

Cemplicity has worked with public health sector clients for a number of years and understands the critical importance of data security when managing patient contact details. Its approach to security touches every part of the company, from personnel management to hosting arrangements. Cemplicity’s protocols have been developed in close consultation with government agencies responsible for the protection of patient privacy and the data security of public health records. It is part of the Connected Health network.

In order to participate in the national survey, PHOs, practices and DHBs will be provided with licences to access this funded patient experience system. No additional IT investment is required for PHOs, practices and DHBs to participate in the core national survey. Only email access and an internet browser are required.

The survey uses the same system currently being used for the national inpatient survey in DHBs. This means consistent reporting and the potential to develop an integrated dashboard of patient experience results.

The system electronically reports patient feedback to enable real-time updates to the dashboard reports. This means minimal administration for PHOs and practices, no postal delay and higher quality, more timely data with minimal intervention. Patients requesting contact can be attended to as soon as possible by an appropriate staff member. This approach also allows the one survey to be presented to each patient under the brand of the practice they attended so there is trust and relevance in the invitation to give feedback.

A key requirement from participating practices, via PHOs, is the provision of patient email addresses and mobile phone contact information. The national system will support in situ surveying (via tablets), subject to the completion of the in situ pilot still in progress.

2.7.1 National system flexibility

The national survey is the core question set that all PHOs must administer for a defined group of patients (see below) once a quarter. This allows derivation of results for domains, modules or specific questions at a national, DHB or PHO level each quarter. Provided the methodology and procedures are followed, the findings should be comparable and statistically robust.

The system is able to present the survey in different languages. The need to introduce language options will be assessed by the governance group in consultation with PHOs once the national programme is implemented.

The survey and reporting system is flexible enough to enable PHOs, practices and DHBs to invest in the programme and add their own questions and/or reports. They may also run a more continuous survey process or one-off surveys if they choose. This would involve negotiating an additional local service agreement with Cemplicity. Should PHOs, practices or DHBs opt into a more continuous approach (eg, weekly or fortnightly survey invitations) they would also get access to an integrated case management functionality that is designed to embed a process of continuous improvement.

PHOs, practices and DHBs considering adding new questions or incorporating the national survey in existing local surveys should be mindful that the size of the survey and time required to complete it directly affect the response rate. For this reason, the governance group will be notified if Cemplicity receives a request to amend the quarterly survey. Any such adaption is likely to be timed for after the programme is implemented nationally. This will avoid conflict or confusion with the national requirements.

3. Survey process

3.1 Process chart – interim

Enrolled patient has a consultation during a set sample week each

quarter

Five pilot PHOs extract patient data from participating practices, in accordance with the survey rules

Extract uploaded via secure FTP to national survey provider (as close to day

extracted from practices as possible)

Reminder sent to patients 7 days later

Survey closes after 21 days

During this time, responses can be viewed

updating ‘live’ in the online

system

Unweighted reports available immediately

Weighted reports available 14 days after the survey closes

PHOs, practices, DHBs and the Commission communicate the new survey through a range of methods

Survey emailed or texted, using practice logo/contact details, to the sample of patients with a mobile or email contact in

the extract, the same day as received from the PHO

See survey annual timetable

3.2 Process chart – final (NES) solution

Enrolled patient has a ‘qualifying event’ during the sample week each

quarter

Extract uploaded via secure FTP to national survey provider (as close to day

extracted from practices as possible)

Survey emailed or texted, using practice logo/contact details, to a sample of patients with a mobile or email contact in the

extract, the same day as received from the NES

Reminder sent to patients 7 days later

Survey closes after 21 days

During this time, responses can be viewed updating ‘live’

on the dashboard

Unweighted reports available immediately

Weighted reports available 14 days after the survey closes

Patient data extracted in accordance with survey rules from NES on set date after

sample period end

PHOs, practices, DHBs and the Commission communicate the new survey through a range of methods

See survey annual timetable

Practice captures survey preference for eligible patients in the practice

management system

3.3 Data flows and system access

3.3.1 Interim solution

Step 1: PHOs send extract information to Cemplicity

PHOs each connect via an SFTP client to the SFTP server housed within its own dedicated virtual machine (VM) within the Datacom application server environment. This SFTP server is secured via individual IP addresses within each of the PHOs and is only open through the firewall on the SFTP port. All other traffic is denied.

On connection, each of the PHOs uploads its patient data extract (csv file) for the period.

During the period the surveys are run, an integration application takes this csv file, converts the data to the appropriate format and stores it inside MS SQL server. The csv file is then deleted. Once this data is transformed, invitations can be sent to the patients via email and SMS.

Step 2: Invitations are sent from the application servers to patients

On sending of invitations, a 36-character GUID is created and used to link the patients’ responses to the non-clinical background data provided in the file. For SMS invitations, a code is given which links to this same 36-character GUID.

Patients receive the invitation via email or SMS then complete the survey online via any web based browser, mobile phone or tablet.

Email invitations are all sent from feedback@myexperience.health.nz with each participating practice’s name or logo (if available) and contact information.

Eligibility for a .health.nz domain name is restricted to organisations that deliver health services through registered practitioners. The myexperience.health.nz address is used as a trusted source of correspondence for recipients.

SMS invitations are sent from the number 2333 with the practice name. This mode is more challenging given character limitations for each SMS sent.

Clicking the link in the invitation, URL https://se.myexperience.health.nz?u=<GUID>, takes patients through the firewall over port 80, and into the application server environment. Patients’ answers are stored in the database as they complete each question page (eg, click ‘next’).

Patients who receive a SMS invitation via a non-smartphone can go to www.myexperience.health.nz and enter their unique eight-digit code to complete the survey.

Note: There is no external access directly to the database servers; these are routed through internal IP addresses.

The SFTP server and the application VMs are separate machines.

Step 3: Report users access data

In the third step, users at individual PHOs, practices, DHBs, the Commission and the Ministry of Health access the reporting portal via a 256-bit encrypted https link. These users are authenticated via a log in that utilises a unique key/SALT algorithm. On gaining access to the portal they are routed to the report application server, which displays aggregated data pulled from the SQL server.

Cookies are used sparingly within the reporting application (but not the survey application accessed by patients responding) and are limited to standard settings like group by or sort order. Users’ preferences are remembered via the cookie. If these preferences do not exist, the application simply defaults to the global setting. Cemplicity also uses a cookie for authentication purposes when users log on. Cemplicity does not record behaviour-specific information outside of these standard preferences.

Step 4: Tidy up

Cemplicity runs scheduled clean-up tasks that delete invitation information at agreed times after a period closes. Patient contact information is only retained in the system for as long as needed to send the survey invitation and reminders. All reportable data is therefore anonymous.

The information flows are shown below in Figure 1.

3.3.2 NES long-term solution

Step 1: The Ministry of Health sends extract information to Cemplicity

Patient information to be used in the survey is captured within the Ministry’s NES database. Practices can update the NES database in real time through their patient management system. In order to send patient information to Cemplicity, the Ministry connects via an SFTP client to the SFTP server that is housed within its own dedicated VM within the Datacom application server environment. This SFTP server is secured via individual IP addresses within each of the PHOs and is only open through the firewall on the SFTP port. All other traffic is denied.

On connection, the Ministry uploads the patient data extract (csv file) for the period.

During the period the surveys are run an integration application takes this csv file, converts the data to the appropriate format, and stores it inside MS SQL server. The csv file is then deleted. In the event a PHO wishes to mail surveys to patients, a new file is put back on the SFTP server for the PHOs to download, which contains the mail sample contact details.

Once this data is transformed, invitations can then be sent to the patients via email and SMS.

Steps 2–4 are the same as described in 3.3.1.

The information flows are shown below in Figure 2.

3.3.3 In situ solution

The in situ process is available to all practices but may especially be valuable to practices who service communities with lower access to email, internet or affordable data plans on their phones. The tablet can be connected via wifi, SIM card or a hotspot device (meaning one SIM card can provide access for a number of tablets).

Tablets need to be set up with a survey link by Cemplicity. The survey link is specific to the practice with the tablet. The practice logo is displayed on the survey. This allows for the practice name to be recorded against survey responses as well as mapping the responder’s PHO and DHB.

Each practice is assigned a 36-character GUID link. Tablets placed inside practices then have the 36-character GUID link added to the tablet. A patient taking the patient experience survey clicks the link and is directed to the survey.

On clicking on the in situ link, the URL https://se.myexperience.health.nz?u=<GUID> will take patients through the firewall over port 80 and into the application server environment. Patients’ answers are stored in the database as they complete each question page (eg, click ‘next’). No survey response data is saved on the tablet. As a patient clicks from page to page, the survey item is saved to the Cemplicity server.

In situ surveying is somewhat simpler because no contact details have to be captured. To prevent patients who complete the survey in situ from also completing the survey if they receive a SMS or email invite, it is recommended reception staff alert patients that this might happen and recommend they ignore the survey invite.

Steps 3–4 are the same as described in 3.3.1.

Figure 1: Data flows interim solution

Figure 2: Data flows NES solution

3.4 Collection method The primary care survey information is collected online. This differs to the adult inpatient survey where some paper collection is involved; however, the sample size per DHB for the inpatient survey is much smaller, being limited to 400 patients per quarter. Online collection is the cheapest, quickest method and enables real-time updates to online reports for practices and PHOs. Patients selected for surveying will be contacted by email (as a preference due to no cost and higher response rate), then SMS (text) and provided with a unique online survey link to click on or type into an internet browser to submit their responses. Email and SMS invitation processes can be augmented in some practices with tablets (in situ) as outlined above.

Once the patient’s unique link closes, there is no patient-identifiable information connected with it so responders are anonymous unless they choose to provide their contact details. This option is provided in case the patient wishes to discuss an issue with the practice (the practice will receive an email alert). Once the survey has been completed, the link expires so there is only one response per patient. All online links expire three weeks from the time the survey is sent.

According to Statistics New Zealand, 80 percent of New Zealanders7 have residential internet access, with 1.6 million connections, while mobile phone internet connections accounted for 3.9 million in 2015. Internet access continues to rise nationwide.

Absence of internet access is concentrated among poorer populations, rural communities and people aged 75 and older, some of whom may be high users of hospital services. In situ surveying is the most promising method to address these audiences.

3.4.1 Pilot analysis

Our pilot analysis of over 1000 respondents has shown survey response online is straightforward and about two-thirds of people complete the survey within 20 minutes. Some people complete the survey in multiple sessions, which takes longer. Some people, albeit with no free text responses, completed the survey in less than five minutes.

The more modules of the survey completed, the longer the response time. The survey contact method (email, SMS, in situ) had little effect on the response time.

There was no relationship between time spent answering the survey and the positivity of responses. Nor was there a relationship between survey contact method and positivity of responses. The process of completing the survey did not appear to affect the content of responses.

Female and middle-aged people were more likely to respond. Asian and European peoples were overrepresented while Māori and Pacific peoples were under-represented. Providing in situ sampling in practices with high Māori and Pacific populations is one way to counteract this.

7 www.stats.govt.nz/~/media/Statistics/browse-categories/industry-sectors/information-technology-communications/ISP-survey/isp15-alltables.xlsx

3.5 Sampling method

3.5.1 Email notification and online collection

The PHO and NES run the patient data extract for the one-week period in accordance with the data extract and file format rules. The file is uploaded through a secure portal to Cemplicity on the same day. Cemplicity will then select all patients with an email address and send the survey invitations on the same day. If the patient has not completed the survey within seven days, they will receive a reminder. After 21 days from the original send date, the survey link will expire.

3.5.2 SMS notification and online collection

Any patient records without an email address but with a mobile phone number will be sent an SMS invitation to complete the survey. If the patient has a smartphone, they can click on the link to complete the survey on their phone. If they have a standard mobile phone they can go to the web page www.myexperience.health.nz in any device’s internet browser, enter a short code (that is in the SMS message) and complete the survey online. If the patient has not completed the survey within seven days, they will receive a SMS reminder. After 21 days, the survey link will expire.

3.5.3 In situ collection

In situ collection is currently being piloted and shows promising results for under-represented populations. In areas where there is low access to the internet, online survey delivery presents a substantial risk of unrepresentative and/or very low response rates. In these areas, practices or PHOs should consider an in situ alternative for the collection of data.

Potential negatives of in situ reporting may include the following.

• The power/mana imbalance between patient and professional may inspire the patient to give unrealistically positive responses.

• Patients seeking assistance with either the device or completing the survey. • Cost of tablets. • Interruption preventing completion.

The process for in situ survey collection is as follows.

1. Discussion between each PHO and Cemplicity to set up the in situ scenario. This tends to be bespoke for each practice/PHO in light of IT policy considerations.

2. Agreement of technical requirements and method for transmitting the data to the national system needs to be made with Cemplicity before undertaking this method.

3. Frontline staff guidance from the Commission. 4. Tablets are set up by Cemplicity and sent to the practice or PHO. 5. Data must be collected in a consistent manner for all patients surveyed so that:

• any eligible patient is offered a tablet to complete the survey (not just friendly, positive patients)

• the member of staff administering the survey does not stay with, advise or otherwise prompt the person taking the survey

• the survey is physically completed by the person taking it, or a carer with them. Members of staff administering the survey cannot complete the survey for patients.

3.6 Frequency of collection For national reporting purposes and consistency, the survey will be sent to patients seen within the same one-week period in each quarter in each participating practice. The one-week period chosen avoids public holidays, but it is not intended as a representation of the quarter in which it sits. Rather, it will be one of a series of snapshots recorded over the course of a year. This fits with the Commission’s recommendation that PHOs collect patient feedback constantly to monitor changes and stimulate improvement in patient experience.

The timetable for the survey is shown in Table 3.

Table 6: Indicative survey timetable

2015/16 2016/17

Process Q3

Jan-Mar 16 Q4

Apr-Jun16 Q1

Jul-Sep16 Q2

Oct-Dec16 Q3

Jan-Mar17 Q4

Apr-Jun17 Patients with a ‘date of last consultation’ at the practice they are enrolled with, in a set one week sample period each quarter

10-16 February

2016

2-8 May 2016

8-14 August 2016

21 October – 6 November

2016

6-12 February

2017

1-7 May 2017

PHO extracts patient data 24-Feb-16 18-May-16 24-Aug-16 16-Nov-16 22-Feb-17 17-May-17

Survey emailed or text to all patient with an email or cell contact

24-Feb-16 18-May-16 24-Aug-16 16-Nov-16 22-Feb-17 17-May-17

Real time, unweighted reports available for the quarter

24-Feb-16 18-May-16 24-Aug-16 16-Nov-16 22-Feb-17 17-May-17

Reminder email or text sent seven days later 02-Mar-16 25-May-16 31-Aug-16 23-Nov-16 01-Mar-17 24-May-17

Email and text survey links close have twenty-one days

16-Mar-16 08-Jun-16 14-Sep-16 07-Dec-16 15-Mar-17 07-Jun-17

Survey response data files provided to PHOs 18-Mar-16 10-Jun-16 16-Sep-16 09-Dec-16 17-Mar-17 09-Jun-17

Draft weighted reports provided to the Ministry, DHBs and PHOs by the Commission

25-Mar-16 24-Jun-16 23-Sep-16 23-Dec-16 27-Mar-17 23-Jun-17

Final weighted reports provided by the Commission

31-Mar-16 30-Jun-16 30-Sep-16 30-Dec-16 31-Mar-17 30-Jun-17

3.7 Eligible patients The survey is designed to be answered by enrolled patients aged 15 and older who have had a consultation (defined per the PHO services agreement) with the practice they are enrolled with in the one-week sample period. Full details of the patient data extract, including rules for inclusions and exclusions, are set out in Appendix 3.

In order to use the online system effectively, email addresses and mobile phone numbers need to be collected. The Commission recommends that email addresses and phone

numbers are captured in a consistent format to help make administration of the survey as simple and cost effective as possible.

3.8 Sampling Adequate sample size is essential so the recorded results represent a population as a whole. At least three risks need to be addressed:

• too small a sample of responders, leading to excessively wide confidence intervals and unstable results, which make changes hard to interpret

• a sample unrepresentative of the population structure • response bias (ie, responders with unrepresentative opinions – more or less content with

their experiences than the average – being more likely to respond).

Well-designed samples can mitigate these risks.

3.8.1 Sample size

The Commission’s approach for the primary care survey is essentially a census seeking to get as many responses as possible among all eligible contactable patients. National reports that use weighted statistics will be restricted to PHOs.

The Commission aims deliberately to over-sample high users of health services, which is why this survey is given to all eligible8 adults receiving services, rather than simply the enrolled population. The survey asks questions about all interactions with health services in the last year, rather than just the most recent GP or nurse appointment.

3.8.2 Reminders

Evidence shows that a reminder will generate one-third to a half of the responses the original survey contact achieves. For this reason, the Commission includes a reminder seven days after the initial contact of the patient.

During the pilot phase, the Commission tested sending a second reminder but this generated negative feedback from patients who were ‘sick of receiving texts’.

Patients are captured in the survey invitation extract if they have attended a practice in the past week. Invitations are then sent promptly on receipt by Cemplicity. This aims to minimise the risk of the patient’s circumstances having changed, and this may include their death. It can take up to three months for systems to be updated following a patient’s death.

While there is no way to entirely remove the risk of sending a survey or reminder to the family of someone who has died, it can be mitigated by minimising the timeframes.

3.8.3 Sampling method

Regardless of sample size, if the sample is systematically unrepresentative of the population the results will be misleading.9 There are two ways to mitigate this: either stratify the sample

d Patients enrolled with and seen by participating practices in the survey sample week each quarter will receive a survey invitation via email or SMS. Children under 15 will not be surveyed. 9 Famously, the Literary Digest miscalled the 1936 US election as a victory for the Republican challenger Albert Landon, despite a sample size of over 2 million, at least in part because of using a telephone directory as the basis of its sample at a time when telephones were a luxury item. http://issuu.com/chilesoc/docs/why-the-1936-literary-digest-poll

so the sample looks like the overall population, or use a random sample and apply weights in line with the local population to the results post hoc.

The Commission has found the second approach more practical in this instance.10 Weighting according to demographic characteristic (age, gender and ethnicity are captured in the survey) allows results to be adjusted to reflect recorded differences. This is an approach used widely by opinion polling organisations.

3.8.4 Demographic weighting

Weighting is a relatively straightforward process which takes either a local or national population structure, compares this with the sample structure, and creates a co-efficient which is applied to the results of the survey. This then increases or decreases a particular score and provides a weighted result. This approach is distinct from standardisation. We are not seeking to compare PHOs or DHBs with each other using this method. Rather we are seeking to weight so that results accurately reflect the views of a representative local population inside a specific PHO or DHB.

The national system will show un-weighted results online for local purposes. Once a sufficient number of practices are participating in the survey, the Commission will prepare quarterly patient experience indicator reports that will show weighted results in accordance with the method described in this document. This will be published on the Commission’s website in the same manner as the adult inpatient experience survey.

The issue that needs to be considered is what variables should be used in the weighting. The primary care survey collects data on age, ethnicity and gender. The Commission has analysed results from the early rounds of surveying by these different groupings to identify whether there are systematically difference results for different populations. The results are quite complex. Age is associated with different responses for all domains with the three age groups aged 65 and over being consistently more positive than working age adults. However, for both gender and ethnicity only the coordination domain and coordination barriers to care sub domain show differences with men and Europeans being more positive. There are no significant differences between other ethnic groups.

In order to have a manageable weighting method across the three groups we have aggregated the groups as follows. Gender is aggregated to male, female and other; age is aggregated to 15-64 and 65+ and ethnicity to European and other. The reason for these groupings is to avoid distortion of the results caused by a large array of “cells” in the weighting calculations with very small numbers of respondents in each one.

To reiterate, the national report does not report results for sub populations, the local reports provide this information in an entirely accurate, unweighted manner.

The notional response of a particular PHO will demonstrate how this base population can be used to create weightings. Table 6 shows PHO A has 1500 respondents to the survey and are divided into groups as follows

10 For stratified approach to completely address this risk, the Commission would have to assume that each strata had a similar response rate and, if they did not, a further exercise in weighting would be required.

Table 7: Demographic make-up of PHO: Survey respondents by age/gender/ethnicity

Gender Age Ethnicity Respondents % Female 15-64 European 491 32.7%

Other 235 15.7%

65+ European 215 14.3%

Other 29 1.9% Male 15-64 European 200 13.3%

Other 89 5.9%

65+ European 203 13.5%

Other 28 1.9% Other 15-64 European 4 0.3%

Other 5 0.3%

65+ European 1 0.1% Other 0 0.0%

The proportion in each group can be compared with the sample population inside the PHO (Table 8) which shows the variance between the survey response and the notional population. Some differences become immediately apparent. There is an over-representation of younger non-European women and older European women, while younger non-European men and older European men under-representation of men in general. This is addressed through weighting. The population proportions are divided by the responder proportions to create a weighting factor for each sub-group, as shown in Table 9 (to avoid individual sub-groups with very small numbers having too much weight in the final weighting, weighting factors are limited to a maximum of five).

Table 8: Demographic make-up of PHO A survey respondents and overall population by age/gender/ethnicity

Gender Age Ethnicity %

responders % population Female 15-64 European 32.7% 33.6%

Other 15.7% 13.9%

65+ European 14.3% 13.4%

Other 1.9% 2.6% Male 15-64 European 13.3% 12.7%

Other 5.9% 6.9%

65+ European 13.5% 14.3%

Other 1.9% 2.0% Other 15-64 European 0.3% 0.3%

Other 0.3% 0.2%

65+ European 0.1% 0.0% Other 0.0% 0.0%

Table 9: Weighting factors by age/gender for PHO A

Gender Age Ethnicity Weighting coefficient

Female 15-64 European 1.03

Other 0.89

65+ European 0.94

Other 1.36 Male 15-64 European 0.95

Other 1.16

65+ European 1.05

Other 1.08 Other 15-64 European 1.08

Other 0.74

65+ European 0.61 Other 0.00

These factors can then be applied to individual responses received to provide weighted scores for individual questions at each DHB. The effect of this is to give different values to responses effectively reflecting how many patients of a different age, gender and ethnicity each respondent is representing. The more over-represented a particular group among the responders, the fewer total patients each responder represents and thus the response is down-weighted and vice versa.

3.8.5 Non-response bias

The inpatient survey is collected primarily online with sampled patients contacted by email, letter and text. Typical completion rates are around 40 per cent for letter, 35 per cent for email and 15 per cent for text with a consistent response rate of approximately 27 per cent. This low response rate has led some to question whether the survey responses can be considered valid and reliable. To address this concern, the Commission worked in collaboration with Cemplicity and the Health Economics Department of Victoria University to investigate further.

The issue of “low response rate” involves three related but distinct issues. First, the number of responses can be so low that results are unreliable. In practical terms this means having confidence intervals so wide that it is nearly impossible to compare one question with another (ie, identifying priority areas for improvement) or the same question over time (ie, identifying improvement over time). Strictly, this is not an issue about response rate. If the sampled population is large enough in the first place, there will be sufficient respondents to avoid this even if the response rate itself is low.

Second, the low response could be because some demographic groups are under-represented, leading to a non-response bias. If these groups hold substantially different opinions and have substantially different experiences, then results will be invalid as they do not represent the whole population. In contrast to the first issue, this pitfall is not addressed by increasing sample size. The risk of unrepresentative results can, to some extent, be mitigated by weighting of responses against the population make up (up-weighting responses from respondents belonging to underrepresented groups and vice versa). This process has its limitations: if numerous demographic characteristics are used to weight for a small sample, individual responses are able to dramatically distort results.

Finally, the low response rate could lead to invalid results if responders systemically held different opinions or had different experiences from non-responders, independent of their demographic makeup. Theoretically, responders could be systematically more positive than non-responders because the latter wish to forget their negative experience. Conversely, responders could be more negative as people are “more likely to respond if they have something to complain about”. Ironically, both these conflicting explanations have been put forward to suggest that the inpatient survey may be invalidated by its response rate.

The study concluded that the non-responders to the original survey gave responses to the follow-up survey that are similar to the responders to the original survey. This suggests putting more effort into increasing the response rate of the Patient Experience Survey is unlikely to change its conclusions. The Commission will publish the final results of this study on its website by June 2017.

3.9 National system provider services The services that the national survey and reporting system provider, Cemplicity, has agreed with the Commission are as follows:

Development work

a. Provide a draft survey online for the online cognitive testing process, working with the appointed cognitive testing provider.

b. Become part of the Connected Health network. c. Import the patient data extracts to the data warehouse initially for up to 6 PHOs and

create a survey invitation list based on sampling criteria specified by the Commission notwithstanding that the sample criteria will be able to be applied using existing Supplier functionality and will not require significant data manipulation.

d. Create real time dashboards to allow reporting of the results, in accordance with the methodology prescribed. Users will be able to filter results and download reports in multiple formats.

e. Create log in access for up to two users at each PHO, two users at each DHB and twenty national users which they will then manage themselves to reflect changes in required staff access.

f. Create the initial log in access for each general practice and provide these details to the practice’s PHO to be distributed to the practices by the PHO. Provide instructions and functionality to each PHO so they can add and delete users at their PHO and practices.

g. Set up the in situ process for piloting.

Ongoing work

a. Import the patient data extract(s) to the data warehouse and apply any stipulated sampling rules.

b. Contact the sample of patients using, in order of priority: email then SMS, and invite participation in the survey using the invitations that are shown in Appendix 4.

c. Ensure that the branding, email signature and survey conclusion signature viewed by each respondent to the electronic survey match the branding of the general practice at which they are enrolled, as informed by the data extract.

d. Provide PHOs that opt to administer mail surveys (this will not be a national requirement) a data file via the secure file transfer process, in accordance with the adult inpatient process. For clarity, the Commission would be responsible for establishing the paper survey process with any PHO wishing to do this.

e. Send electronic reminders to complete the survey seven days following initial contact using the same method as initial contact.

f. Surveys will remain open for 21 days. After each survey is closed, all personally identifiable information in the data extracts that were provided by the PHOs or NES will be permanently deleted from Cemplicity’s system. (Cemplicity will not be able to reactivate survey links once this information is deleted if respondents wish to provide responses after the survey links are closed.)

g. Send an email alert to a nominated practice email address when a respondent indicates they wish to be contacted by the practice. Provide contact details and reason for contact to the practice for those patients who wish to discuss some aspect of their response.

h. Provide a survey data extract to PHOs and DHBs (as deemed appropriate by the governance group) for local analysis on request up to once per quarter in the format prescribed in the methodology document, in accordance with agreed data access rules.

i. Provide a national extract (or a simple user interface to enable the Commission to extract data as required) at least quarterly, to allow calculation of the national indicators. The Commission will be able to access and download national data at any time, in accordance with agreed data access rules.

j. Conduct the above processes in accordance with the national timetable. k. Data received from PHOs or from NES should be validated (batch files and records) with

error reports sent back to the data provider.

All data will be hosted and stored in New Zealand unless agreed in writing by the Commission and the Health IT Board that an alternative hosting arrangement is acceptable.

Further information on the Services agreement can be sought from the Commission, however this document is commercially confidential and any information provided needs to be done so with Cemplicity’s consent.

4. Use of the survey results

4.1 National accountability The survey is designed to encourage local improvement and provide national indicators by DHB or PHO for each of the domains. This will consist of the weighted average of the 0 - 10 rating. The weighting will apply the factors set out in section 3.8.4 above. To allow comparability, this will be based on the national population of eligible patients (this will be tested as the survey develops. This isn’t in the contract and needs more thought to be sure it is a good idea).

4.1.1 Calculation of system level measures

Each responder is given a weighting factor based on the method set out at 3.8.4. The 0 - 10 rating given by the individual is multiplied by the weighting factor to give a weighted rating. These are then summed and divided by the total number of responders to give a weighted average.

The following shows a worked example using the notional “PHO A” survey responders set out in section 3.8.4.

Step 1: find ratings for each individual

Respondent Gender Age group

Communication rating (0-10)

1 Female 15-44 9 2 Female 15-44 10 … … … … 390 Male 85+ 9 391 Male 85+ 7 Average 7.45

Step 2: apply appropriate sub group weighting for each individual

Respondent … Communication

rating (0-10) Weighting

factor 1 … 9 x 0.9145 2 … 10 x 0.9145 … … … … … 390 … 9 x 1.0455 391 … 7 x 1.0455 Average … 7.45

Step 3: calculated weighted average

Respondent … Communication

rating (0-10) Weighting factor Weighted

rating 1 … 9 X 0.9145 8.2305 2 … 10 X 0.9145 9.145

… … … … … 390 … 9 X 1.0455 9.4095 391 … 7 X 1.0455 7.3185 Average … 7.45 Weighted Average 7.54

In this worked example, an arithmetic average rating of 7.45 increases to 7.54 once the weighting factors are applied.

This calculation will be applied to each rating for each DHB for each national quarterly snapshot in order to provide a weighted indicator.

4.2 Online reports All practices, PHOs and DHBs have the same national standard report view (subject to the data access matrix) when they log in to the national survey and reporting system. Appendix 7 shows each of the main report types.

4.2.1 Accessing the survey results

Type https://cx.myexperience.health.nz/pes in an internet browser.

Enter email and password and click the green ‘log in’ button.

4.2.2 The dashboard

Once logged in, the user will arrive at the ‘dashboard’. This landing page provides a snapshot of key metrics at a national level.

4.2.3 Opening the reports

Below is the basic report layout showing the three main elements of the screen.

Report menu The report menu allows users to navigate from the main dashboard to more detailed reporting views. The report menu aligns closely with the structure of the patient experience survey to give the user a quick reference point for evaluating patient response.

Graphs and tables Across the reporting portal, different types of graphs are carefully selected and used to present the data in the clearest format possible. For example, bar charts are used to give a clear view of trends from quarter to quarter.

Filters The filters allow users to isolate responses by age, gender or ethnicity for in-depth comparison between general practices, PHOs, DHBs and the national average. For example, using the ethnicity filter the user can compare how Māori responses vary from non-Māori responses in relation to GP access between PHOs.

4.2.3 Viewing reports

Several types of reporting in the patient experience survey can be found by clicking on each heading within the report menu. Note: the view you see will depend on whether you are from a practice, a PHO or DHB.

Trend reports These reports are designed to show how the user’s ratings are changing over time.

By PHO These reports allow you to compare patient feedback across PHOs.

By lead DHB These reports allow you to compare patient feedback across DHBs.

Comments These reports present the comments patients have given in the survey. The comments may be moderated for the recommended process. (For more information on how to do this read the full portal user guide – link below.)

4.2.4 Printing reports

You can print reports by clicking on the ‘Print’ button to the top right of each graph. This will generate an image of the report, along with any filters you currently have applied. Alternatively, you can click on the three horizontal lines, on each graph just below the ‘response count’ section. The options in this list will allow you to export the graph by itself, without any other information.

The full system manual is available by going to the reporting portal (https://cx.myexperience.health.nz/pes) and clicking on ‘View the Reporting Guide’ under ‘Useful Links’.

4.3 Quantitative and qualitative analysis Cemplicity will provide a survey data file in accordance with Appendix 6 to PHOs and the Commission so that further analysis can be performed.

PHOs are able to further analyse or report the patient survey information as they choose for internal purpose. In the patient comments reports users are able to filter and view word clouds, and search on particular terms. Further analysis can be undertaken by PHOs using the data extract provided.

The Commission will take a longer term view of the information and perform analysis as required to look at differential response, response rates and identify national trends that may suggest changes to the survey tool.

4.4 Continuous improvement The patient experience survey is about improving the quality of health services in New Zealand by enabling patients to provide feedback that can be used to monitor and improve the quality and safety of health services.

Being able to capture, understand and act on patient experiences in a timely manner is a vital contributor to the improvement of health service delivery and the prioritisation of attention and resources.

A critical element of this programme is the delivery of value to practices, PHOs and DHBs, while meeting the national reporting needs. Practices, PHOs and DHBs need a programme that is simple for them to participate in, one which provides actionable information for improvement activities and enables prompt follow up with patients requesting contact.

Key to enabling this continuous improvement is the decision to develop an electronic system that captures and reports patient feedback on a timely basis. Enabling patients to support their ratings with comments and examples lets practices and PHOs know what is being done well and what needs to change. The online methodology is effective at capturing rich patient stories as people can complete the survey when and where it suits them and without the presence of an interviewer.

5. Annual review

Changes to the primary care patient experience survey, the reporting system and the methodology and procedures document will be made on an annual basis. This is to minimise the impacts on trending reports, process changes and costs.

Following the first evaluation process described in 1.3.5, an annual evaluation of the programme will be conducted by the Commission to ensure continuous improvement. As part of this annual evaluation Cemplicity will survey all system users to understand the user experience with the tools, how the system is being used and outcomes at practice, PHO, DHB and national levels.

The evaluators would provide the questions to be asked of users through the survey to the governance group. At this time we would agree how the programme could be enhanced to lead to better patient safety and quality of health service delivery.

Appendix 1: Primary care patient experience survey

Survey overview

1. This version of the survey does not show the logic that ‘skips’ people to appropriate questions based on their answers. Not all people see all the questions.

2. Questions in blue are the main logic drivers of the survey. Respondents will only be asked about things they have indicated are applicable to them. This is represented by the flower image to the right, whereby all patients see questions about their regular GP and place of care but only see other questions (the petals/modules) if they are relevant to the patient’s experience.

3. All open comment questions are optional for respondents.

Introduction

Could you tell us if you are answering this survey on behalf of yourself or someone else? Single selection (radio buttons) 1. Myself 2. Someone else (Please tell us why) GP clinic or community health clinic

This section has a few questions about your GP clinic or community health clinic.

Is there one GP clinic or community health clinic you usually go to? Single selection (radio buttons) 1. Yes 2. No, I do not have a place that I usually go Next question only applicable to people that answer ‘No’ to the previous question

You are enrolled at [name of GP clinic or community health clinic]. Is there a reason you don’t usually go there? Large text area Does the place you usually go to have a phone service, doctor or nurse available during evenings, nights or weekends? (Either there or another place) Single selection (radio buttons) 1. Yes 2. No 3. Don't know

Is there one GP or nurse you usually see? Single selection (radio buttons) 1. Yes 2. No How long have you been going to your current GP or nurse clinic? Single selection (radio buttons) 1. Less than one year 2. One to five years 3. More than five years Please answer this next section based on your experiences over the last 12 months

When you ring to make an appointment how quickly do you usually get to see... Matrix (radio buttons) Top options 1. Same day 2. Next working day 3. Within a week 4. Over a week 5. Not applicable Side options 1. Your current GP? 2. Any other GP at the clinic you usually go to? 3. A nurse at the clinic you usually go to? Is this acceptable? Single selection (radio buttons) 1. Yes 2. No (Please tell us why): How long do you usually have to wait for your consultation to begin with... Matrix (Radio Buttons) Top options 1. 5 minutes or less 2. 6–15 minutes 3. 16–30 minutes 4. More than 30 minutes 5. Not applicable Side options 1. Your current GP? 2. Any other GP at the clinic you usually go to? 3. A nurse at the clinic you usually go to? Is this acceptable? Single selection (radio buttons)

1. Yes 2. No (Please tell us why): Did the reception and admin staff treat you with respect? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No In the last 12 months was there a time when you did not visit a GP or nurse because of cost? Single selection (radio buttons) 1. No 2. Yes (Please tell us why): Could you tell us why cost stopped you from seeing a GP or nurse? Multiple selection (checkboxes) 1. The appointment was too expensive 2. The cost to travel was too expensive 3. I couldn't afford to take the time off work 4. Other (Please tell us): Was there ever a time when you wanted health care from a GP or nurse but you couldn’t get it? Single selection (radio buttons) 1. No 2. Yes (Please tell us why): When you contact your usual GP clinic about something important, do you get an answer the same day? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Does your GP or nurse explain things in a way that is easy to understand? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Are you confident that your GP or nurse is aware of your medical history? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes

3. No 4. Don’t know Have you been involved in decisions about your care and treatment as much as you wanted to be? Single selection (radio buttons) 1. Yes 2. Yes, to some extent 3. No Does your GP or nurse... Matrix (radio buttons) Top options 1. Yes, always 2. Yes, sometimes 3. No Side options 1.Treat you with respect? 2. Treat you with kindness and understanding? 3. Listen to what you have to say? 4. Spend enough time with you? Is there anything you would like to tell us about your experience with your GP or nurse? Large text area

Now we ask you some general questions so we know which topics to ask about later in the survey

Do you take any medication regularly? This includes vitamins, painkillers, supplements and any prescribed medication. Single selection (radio buttons) 1. Yes 2. No In the last 12 months... Matrix (radio buttons)

Overall, was your experience with your GP or nurse clinic? (Please select a number) Very poor Excellent 0 1 2 3 4 5 6 7 8 9 10

Medication

The next questions are about medication, including vitamins, painkillers, supplements and any prescribed medication you have used during the last 12 months. From now on we’ll refer to these as ‘medication’.

Were you involved as much as you wanted to be in decisions about the best medication for you? Single selection (radio buttons) 1. Yes, definitely 2. Yes, to some extent 3. No 4. I did not want to be involved Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). Matrix (radio buttons) Top Options 1. Yes definitely 2. Yes to some extent 3. No 4. N/A Side options 1. Was the purpose of the medication properly explained to you? 2. Were the possible side effects of the medication explained in a way you could

understand? 3. Were you told what could happen if you didn’t take the medication, in a way you could

understand? 4. Were you told what to do if you experienced side effects?

Did you have any tests such as x-rays, scans, blood tests or other tests?

Yes No Don't know

Have you seen any health care professionals other than a GP or nurse? Some examples are a midwife, physiotherapist, psychologist, social worker, counsellor, pharmacist, or dietitian.

Yes No Don't know

Have you seen a specialist doctor, other than a GP?

Yes No Don't know

Have you been to the emergency department at the public hospital?

Yes No Don't know

Have you stayed in hospital overnight?

Yes No Don't know

Do you have a health condition that will last more than six months?

Yes No Don't know

Did you follow the instructions when you took the medication? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No You said that you did not always follow the instructions when you took the medication. Please tell us why. Multiple selection (checkboxes) 1. Cost 2. Side effects 3. I forgot 4. I felt better 5. Other (Please tell us more): Has cost stopped you from picking up a prescription? Single selection (radio buttons) 1. No 2. Yes In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist (outside of hospital)? Single selection (radio buttons) 1. No 2. Yes 3. Don't know Because of the wrong medication or dose, did you... Matrix (radio buttons) Top options 1. Yes 2. No Side options 1. Stop taking it? 2. Get medical advice? 3. Get medical care? 4. Get admitted to hospital? Is there anything else you’d like to tell us about being given the wrong medication or dose? Large text area Is there anything you would like to tell us about your experiences with your medication? Large text area

Medical tests

Now a few questions about medical tests and scans you may have had in the past 12 months. This includes x-rays, scans, blood tests and other tests.

Was the need for the x-ray, test or scan(s) explained in a way you could understand? Single selection (radio buttons) 1. Yes, completely 2. Yes, to some extent 3. No 4. I did not need an explanation Were you told how you could find out the results of your x-ray, scan(s) or tests? Single selection (radio buttons) 1. Yes 2. No 3. Not sure 4. I did not need an explanation Were the results of the x-ray, test or scan(s) explained in a way you could understand? Single selection (radio buttons) 1. Yes, completely 2. Yes, to some extent 3. No 4. Not sure 5. I was told I would get the results at a later date 6. I was never told the results of the tests Is there anything you would like to tell us about your experiences with x-rays, scan(s) or tests? Large text area

Other health care professionals

Now a few questions about health care professionals (other than a doctor or nurse) you may have seen or talked to.

Some examples are midwife, physiotherapist, psychologist, social worker, counsellor, pharmacist, and dietitian.

Was there a time when test results or information was not available at the time of your appointment with the health care professional? Single selection (radio buttons) 1. No 2. Yes 3. Don't know 4. N/A Were you given conflicting information by different doctors or health care professionals, e.g. one would you tell you one thing and then another would tell you

something different? Single selection (radio buttons) 1. No 2. Yes, sometimes 3. Yes, always How quickly do you usually get to see a health care professional? Single selection (radio buttons) 1. Less than a week 2. 1–4 weeks 3. 1–3 months 4. Longer Has cost stopped you from seeing a health care professional? Single selection (radio buttons) 1. No 2. Yes Could you tell us why cost stopped you from seeing a health care professional? Multiple selection (checkboxes) 1. The appointment was too expensive 2. The cost to travel was too expensive 3. I couldn't afford to take the time off work 4. Other (Please specify): Is there anything you would like to tell us about your experiences with health care professionals (other than a doctor or nurse)? Large text area Specialist doctors (other than GP)

Now a few questions about specialist doctors (other than GPs) that you may have seen in the past 12 months. When you were referred to a specialist did you have any difficulties getting an appointment? Single selection (radio buttons) 1. No 2. Yes Any comments? Large text area In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor? Single selection (radio buttons) 1. Less than a week 2. 1–4 weeks

3. 1–3 months 4. Longer Any comments? Large text area Has cost stopped you from seeing a specialist doctor? Single selection (radio buttons) 1. No 2. Yes Could you tell us why cost stopped you from seeing a specialist doctor? Multiple selection (checkboxes) 1. The appointment was too expensive 2. The cost to travel was too expensive 3. I couldn't afford to take the time off work 4. Other (please tell us): When you received care or treatment from specialist doctors, did they do the following? Matrix (radio buttons) Top options 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Side options 1. Ask what is important to you? 2. Tell you about treatment choices in ways you could understand? 3. Involve you in decisions about your care or treatment as much as you wanted to be? Any comments? Large text area Do the specialist doctors know your medical history and the reason for your visit? Single selection (radio buttons) 1. Yes, fully aware 2. Yes, aware in part 3. No 4. Don't know Has a doctor ordered a test (e.g. blood test, x-ray, etc) that you felt you didn’t need because the test had already been done? Single selection (radio buttons) 1. No 2. Yes 3. Don't know 4. N/A

Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. Don't know 5. N/A Is there anything you would like us to know about how well your GP and specialist doctors are working together? Large text area Emergency departments

The next questions are about accessing health care from a public hospital emergency department.

The last time you went to the public hospital emergency department, why did you go there? Multiple selection (checkboxes) 1. It was clearly an emergency 2. I was told to go to the emergency department by a health care professional 3. I can’t afford to go anywhere else 4. Other (Please tell us why): Is there anything you would like us to know about how well your GP and the emergency department are working together? Large text area

Hospital stays

The next questions are about your most recent stay in hospital.

Could you please tell us which hospital you received care from? Single selection (dropdown menu) Dropdown list of hospitals. Did the hospital arrange follow-up care with a doctor or other health care professional? Single selection (radio buttons) 1. Yes 2. No 3. N/A 4. Don't know Did your current GP seem informed and up-to-date about the plan for follow-up? Single selection (radio buttons) 1. Yes 2. No 3. N/A 4. Don't know Did you have to go back to hospital or get emergency care because of complications within a month after being discharged from hospital? Single selection (radio buttons) 1. No 2. Yes Is there anything you would like to tell us about your experience of your GP and the hospital working together? Large text area Long-term conditions

The next questions are about health conditions that are expected to last 6 months or more. These are referred to as 'long-term' conditions.

How long ago were you first diagnosed for the condition(s)? Matrix (radio buttons) Top options 1. Less than 6 months ago 2. 6–12 months ago 3. 1–2 years ago 4. 2–5 years ago 5. 5–10 years ago 6. Over 10 years ago 7. Don't know 8. N/A

Side options 1. Anxiety 2. Arthritis 3. Asthma 4. Cancer 5. Chronic obstructive pulmonary disease (COPD) 6. Depression 7. Diabetes 8. Heart disease 9. High blood pressure 10. Long-term pain 11. Other mental health conditions 12. Stroke 13. Other Which other long-term condition(s) have you been diagnosed with? Large text area Please answer each of the following questions using the column headings as a guide. Matrix (radio buttons) Top options 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Side options 1. Were you given information you could understand about things you should do to improve

your health? 2. Did you get help to make a treatment or care plan for your long-term condition that would

work in your daily life? 3. After a treatment or care plan was made were you contacted to see how things were

going?

Is there anything you would like to tell us about your experience being treated for a long-term condition? Large text area We have just a few quick questions about you to help us better understand your answers

Are you... Single selection (radio buttons) 1. Female 2. Male 3. Gender diverse Please tell us the year of your birth. Single selection (dropdown menu)

Which ethnic group or groups do you belong to? Multiple selection (checkboxes) 1. New Zealand European 2. Māori 3. Samoan 4. Cook Island Māori 5. Tongan 6. Niuean 7. Chinese 8. Indian 9. Other (such as Dutch, Japanese, Tokelauan) or Prefer not to answer You selected 'other' as an option for your ethnic group. Which of these ethnic groups do you belong to? Multiple selection (checkboxes) 1. Other European 2. Tokelauan 3. Fijian 4. Other Pacific Peoples 5. Southeast Asian 6. Other Asian 7. Middle Eastern 8. Latin American / Hispanic 9. African (or cultural group of African origin) 10. Other ethnicity 11. Don't know 12. Prefer not to answer In which language(s) could you have a conversation about a lot of everyday things? Multiple selection (checkboxes) 1. English 2. Māori 3. Samoan 4. New Zealand Sign Language 5. Other language(s), eg, Gujarati, Cantonese, Greek (Please tell us): 6. Would rather not say Did you need an interpreter to communicate with a health care professional? Single selection (radio buttons) 1. No 2. Yes, I had an interpreter 3. Yes, I used a family member as an interpreter 4. Yes, but I did not have an interpreter Any comments? Large text area Did the interpreter help you clearly communicate with the health care professional? Single selection (radio buttons) 1. Yes

2. Yes to some extent 3. No Any comments? Large text area Was cultural support available when you needed it? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. I did not need cultural support Contact request

Would you like someone to contact you to discuss your feedback in this survey? This means you will no longer be anonymous. Single selection (radio buttons) 1. No thanks 2. Yes, I would like someone from my current GP clinic to contact me to discuss my

feedback or health experience Please phone your GP clinic as usual for any medical matters that require a consultation. Are you happy for the person contacting you to see a copy of your survey response? This means that your response will no longer be anonymous. Single selection (radio buttons) 1. Yes, I am happy for them to see a copy of my survey response 2. No, I do not want them to see my survey response – I would like it to remain anonymous Please tell us your contact details. Vertical text box list 1. Your name: 2. Email address: 3. Phone number during the daytime: Please provide some information on what you would like to talk to us about. We can then ensure the right person contacts you. Large text area

Survey copy request

Would you like to be emailed a copy of your survey response? Please note that as this survey is anonymous, if you do not request a copy of your survey response now and you have not requested contact from us, we will be unable to obtain a copy for you in future.

Single selection (radio buttons) 1. No thanks 2. Yes, please email a copy of my survey response to [text box]

Thank you for your time and feedback. You have now finished this survey.

We have recorded all your answers so you can now close this window.

Thanks again,

[Practice Signatory]

Appendix 2: Domain questions and picker scoring

Coordination

Sub-domain Question Text QOptionText Value RC, IC, Seamless Is there one GP or nurse you usually see? Yes 10

No 0

Barriers to access In the last 12 months was there a time when you did not visit a GP or nurse because of cost? No 10 Barriers to access In the last 12 months was there a time when you did not visit a GP or nurse because of cost? Yes (Please tell us why): 0 RC, IC, Seamless Was there ever a time when you wanted health care from a GP or nurse but you couldn’t get it? No 10 RC, IC, Seamless Was there ever a time when you wanted health care from a GP or nurse but you couldn’t get it? Yes (Please tell us why): 0

RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? Yes, always 10 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? Yes, sometimes 5 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? No 0 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? N/A Not scored RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? Yes, always 10 RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? Yes, sometimes 5 RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? No 0 RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? Don’t know Not scored RC, IC, Seamless In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist

(outside of hospital)? No 10

RC, IC, Seamless In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist (outside of hospital)?

Yes 0

RC, IC, Seamless In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist (outside of hospital)?

Don't know 5

Barriers to access Has cost stopped you from picking up a prescription? No 10 Barriers to access Has cost stopped you from picking up a prescription? Yes 0 RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare

professional? No 10

RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare professional?

Yes 0

RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare professional?

Don't know 5

RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare professional?

N/A Not scored

RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn’t need because the test had already been done?

Yes 0

RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn’t need because the test had already been done?

No 10

RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn’t need because the test had already been done?

Don't know 5

RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn’t need because the test had already been done?

N/A Not scored

Barriers to access Has cost stopped you from seeing a healthcare professional? No 10

Barriers to access Has cost stopped you from seeing a healthcare professional? Yes 0

Barriers to access Has cost stopped you from seeing a specialist doctor? No 10

Barriers to access Has cost stopped you from seeing a specialist doctor? Yes 0

RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? Yes, fully aware 10

RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? Yes, aware in part 5

RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? No 0

RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? Don't know Not scored

RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Yes, always 10

RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Yes, sometimes 5

RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? No 0

RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Don't know Not scored

RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? N/A Not scored

RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? Yes 10

RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? Don't know 5

RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? No 0

RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? N/A Not scored

Physical and Emotional Needs

Question Text Question Sub-part QOptionText Value Is this acceptable? Yes 10 Is this acceptable? No (Please tell us why): 0 Did the reception and admin staff treat you with respect? Yes, always 10 Did the reception and admin staff treat you with respect? Yes, sometimes 5 Did the reception and admin staff treat you with respect? No 0 How quickly do you usually get to see a healthcare professional? Less than a week 10 How quickly do you usually get to see a healthcare professional? 1-4 weeks 7 How quickly do you usually get to see a healthcare professional? 1-3 months 3 How quickly do you usually get to see a healthcare professional? Longer 0 In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor?

Less than a week 10

In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor?

1-4 weeks 7

In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor?

1-3 months 3

In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor?

Longer 0

Was cultural support available when you needed it? Yes, always 10 Was cultural support available when you needed it? Yes, sometimes 5 Was cultural support available when you needed it? No 0 Was cultural support available when you needed it? I did not need cultural support Not scored Does your GP or nurse... Treat you with respect? Yes, always 10 Does your GP or nurse... Treat you with respect? Yes, sometimes 5 Does your GP or nurse... Treat you with respect? No 0 Does your GP or nurse... Treat you with kindness and understanding? Yes, always 10 Does your GP or nurse... Treat you with kindness and understanding? Yes, sometimes 5 Does your GP or nurse... Treat you with kindness and understanding? No 0 Does your GP or nurse... Spend enough time with you? Yes, always 10 Does your GP or nurse... Spend enough time with you? Yes, sometimes 5 Does your GP or nurse... Spend enough time with you? No 0 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Same day 10 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Next working day 7 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Within a week 3 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Over a week 0 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Not applicable Not scored When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Same day 10 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Next working day 7 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Within a week 3 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Over a week 0 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Not applicable Not scored When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Same day 10 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Next working day 7 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Within a week 3 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Over a week 0 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Not applicable Not scored

Is this acceptable? Yes 10 Is this acceptable? No (Please tell us why): 0 How long do you usually have to wait for your consultation to begin with... Your current GP? 5 minutes or less 10 How long do you usually have to wait for your consultation to begin with... Your current GP? 6-15 minutes 7 How long do you usually have to wait for your consultation to begin with... Your current GP? 16-30 minutes 3 How long do you usually have to wait for your consultation to begin with... Your current GP? More than 30 minutes 0 How long do you usually have to wait for your consultation to begin with... Your current GP? Not applicable Not scored How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? 5 minutes or less 10 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? 6-15 minutes 7 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? 16-30 minutes 3 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? More than 30 minutes 0 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? Not applicable Not scored How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? 5 minutes or less 10 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? 6-15 minutes 7 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? 16-30 minutes 3 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? More than 30 minutes 0 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? Not applicable Not scored

Communication

Question Text QOptionText Value Does your GP or nurse explain things in a way that is easy to understand? Yes, always 10

Does your GP or nurse explain things in a way that is easy to understand? Yes, sometimes 5

Does your GP or nurse explain things in a way that is easy to understand? No 0

Does your GP or nurse explain things in a way that is easy to understand? N/A Not scored

Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). Yes, definitely 10

Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). Yes, to some extent 5

Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). No 0

Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). N/A Not scored

Was the need for the x-ray, test or scan(s) explained in a way you could understand? Yes, completely 10

Was the need for the x-ray, test or scan(s) explained in a way you could understand? Yes, to some extent 5

Was the need for the x-ray, test or scan(s) explained in a way you could understand? No 0

Was the need for the x-ray, test or scan(s) explained in a way you could understand? I did not need an explanation Not scored

Were you told how you could find out the results of your x-ray, scan(s) or tests? Yes 10

Were you told how you could find out the results of your x-ray, scan(s) or tests? No 0

Were you told how you could find out the results of your x-ray, scan(s) or tests? Not sure 5

Were you told how you could find out the results of your x-ray, scan(s) or tests? I did not need an explanation Not scored

Were the results of the x-ray, test or scan(s) explained in a way you could understand? Yes, completely 10

Were the results of the x-ray, test or scan(s) explained in a way you could understand? Yes, to some extent 7

Were the results of the x-ray, test or scan(s) explained in a way you could understand? No 3

Were the results of the x-ray, test or scan(s) explained in a way you could understand? Not sure Not scored

Were the results of the x-ray, test or scan(s) explained in a way you could understand? I was told I would get the results at a later date Not scored

Were the results of the x-ray, test or scan(s) explained in a way you could understand? I was never told the results of the tests 0

Were you given conflicting information by different doctors or healthcare professionals e.g. one would you tell you one thing and then another would tell you something different? No 10 Were you given conflicting information by different doctors or healthcare professionals e.g. one would you tell you one thing and then another would tell you something different? Yes, sometimes 5 Were you given conflicting information by different doctors or healthcare professionals e.g. one would you tell you one thing and then another would tell you something different? Yes, always 0

Question Text Question Subpart QOptionText Value Does your GP or nurse…… Listen to what you have to say? Yes, always 10 Does your GP or nurse…… Listen to what you have to say? Yes, sometimes 5 Does your GP or nurse…… Listen to what you have to say? No 0 Does your GP or nurse…… Listen to what you have to say? N/A Not scored

When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could understand?

Yes, definitely 10

When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could understand?

Yes, to some extent 5

When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could understand?

No 0

When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could understand?

N/A Not scored

Please answer each of the following questions using the column headings as a guide. Were you given information you could understand about things you should do to improve your health?

Yes, completely 10

Please answer each of the following questions using the column headings as a guide. Were you given information you could understand about things you should do to improve your health? Yes, to some extent 5

Please answer each of the following questions using the column headings as a guide. Were you given information you could understand about things you should do to improve your health? No 0

Please answer each of the following questions using the column headings as a guide. Were you given information you could understand about things you should do to improve your health? I did not need an

explanation Not scored

Partnership

Question Text Question Sub-part QOptionText Value Have you been involved in decisions about your care and treatment as much as you wanted to be? Yes 10 Have you been involved in decisions about your care and treatment as much as you wanted to be? Yes, to some extent 5 Have you been involved in decisions about your care and treatment as much as you wanted to be? No 0 Were you involved as much as you wanted to be in decisions about the best medication for you? Yes, definitely 10 Were you involved as much as you wanted to be in decisions about the best medication for you? Yes, to some extent 5 Were you involved as much as you wanted to be in decisions about the best medication for you? No 0 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? Yes, always 10 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? Yes, sometimes 5 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? No 0 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? N/A Not scored When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or

treatment as much as you wanted to be? Yes, always 10

When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or treatment as much as you wanted to be?

Yes, sometimes 5

When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or treatment as much as you wanted to be?

No 0

When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or treatment as much as you wanted to be?

N/A Not scored

Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life?

Yes, always 10

Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life?

Yes, sometimes 5

Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life?

No 0

Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life?

N/A Not scored

Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you contacted to see how things were going?

Yes, always 10

Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you contacted to see how things were going?

Yes, sometimes 5

Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you contacted to see how things were going?

No 0

Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you contacted to see how things were going?

N/A Not scored

Appendix 3: Patient experience – interim solution – patient data extract

This document describes the rules surrounding the data extract and resulting file format required for importing patients’ information from PHOs into the national system’s data warehouse.

Data extract rules Each PHO extracts the patient data for importing into the PES data warehouse (for sending survey invites, reminders and generating dashboard reports), in accordance with the survey timetable. We highly recommend the process is automated as it means the file format and rules are consistently applied over time and the process can run irrespective of staff availability.

The patient extract should include all patient consults that satisfy the following rules:

1. Frequency – one week each quarter according to the survey timetable. 2. Date Range: for the first survey February 2016, patients with a qualifying event that falls within the 7 day period from Wednesday 10

February 00:01 to Tuesday 16 February 23:59. 3. The extract file should exclude any patients if they have been previously included in an extract file supplied in the last 12 weeks. Exclude

these patients using NHI prior to FTP upload. (This rule needs to be turned off if an extract is being regenerated for some reason) 4. Only include people aged 15 and over – [Consultation date] – [Date of birth] >= 15 years. 5. Include all records even if no email address or mobile number. 6. All deceased patients as at the date the extract is run should be excluded from the extract (do not exclude based on the extract date range). 7. Each patient should only appear once in the extract file – PHO to check for and remove duplicates. 8. Only the patient’s most recent qualifying event within the extraction period should be included (eg, if seen 10 Feb, 12 Feb and 14 Feb, only

include 14 Feb).

File format

The file received from the PHO needs to meet the following criteria:

• The file must be in comma delimited format (conforming to http://tools.ietf.org/html/rfc4180 standard). • All column headings must be provided as the first line and must match the Field name specified in the table below.

• Where required, use ‘0d0a’ as a record terminator. ‘0’ is a zero. • The extract file must be named using the following convention: e.g. PROCARE_PE_From_yyyymmdd_To_yyyymmdd.csv • Codes as opposed to descriptions will be used in columns wherever possible. • DateTime values should be provided using yyyyMMdd hh:mm:ss • When fields have embedded commas, commas need to be placed inside double quotes as per the following example:

“Nick”,”1 Story Road, Otahuhu”, 21, 33, “This is a comment, with a comma”

In the example above, the fields with embedded commas are enclosed in double quotes as per normal CSV rules. This then renders 5 columns for the comma delimited row. If there is an embedded double quote in the field, then it can be escaped by preceding it with another double quote as per the specification.

File transfer We are able to accept secure file transfers using the FTPS or SFTP protocols. While both protocols are secure, we recommend using SFTP to avoid firewall setup issues. The details for connecting to Cemplicity’s server will be given directly to the PHO’s nominated IT contact person. SFTP (Secure Shell FTP) - recommended Requires port 22 to be open on your firewall. FTPS (FTP over SSL) FTPS may be connected using explicit or implicit modes; however explicit is recommended. Explicit mode requires port 21, and passive port range 51500 – 51600 to be open on your firewall. Implicit mode requires port 990, and passive port range 51500 – 51600 to be open on your firewall.

Data fields Field Data type Mandatory

value Allowed options (if restricted)

Example data Comment

NHI Number Alphanumeric - CHB2702 Title / Prefix Text - - Mrs First Given Name Alphanumeric - Jennifer First Preferred Name

Alphanumeric - - Jenny

Family Name Alphanumeric - Smith Mobile phone Alphanumeric - 0279876543 Email address Alphanumeric - david@gmail.com Gender Alphanumeric F

I M U

F Allowed codes from: http://www.health.govt.nz/nz-health-statistics/data-references/code-tables/common-code-tables/sex-type-code-table

Date of birth Date - 19900615 The patient’s date of birth Date of qualifying event

Date - 20110816 Otherwise known as ‘Date of last consultation’ and often the ‘Date of invoice’ field is used

Ethnicity 1 Integer Only those codes in the Level 2 code table.

21 Allowed codes from: http://www.health.govt.nz/nz-health-statistics/data-references/code-tables/common-code-tables/ethnicity-code-tables

Ethnicity 2 Integer - Only those codes in the Level 2 code table.

Allowed codes from: http://www.health.govt.nz/nz-health-statistics/data-references/code-tables/common-code-tables/ethnicity-code-tables

Field Data type Mandatory value

Allowed options (if restricted)

Example data Comment

Ethnicity 3 Integer - Only those codes in the Level 2 code table.

Allowed codes from: http://www.health.govt.nz/nz-health-statistics/data-references/code-tables/common-code-tables/ethnicity-code-tables

HPI-O (practice) Alphanumeric F2N084-H HPI-O (PHO) Alphanumeric F2N084-H PHO Org ID Alphanumeric 794645 DHB of domicile (patient)

Integer DHB Area codes 123 or 011 DHB that the patient is domiciled in. See DHB codes below

Practice DHB Integer DHB Area codes 123 or 011 DHB the practice is physically located in. See DHB codes below.

Lead/PHO DHB Integer DHB Area codes 123 or 011 DHB the PHO holds a contract with. See DHB codes below.

DHB codes 011 Northland 021 Waitemata 022 Auckland 023 Counties Manukau 031 Waikato 042 Lakes 047 Bay of Plenty 051 Tairāwhiti 071 Taranaki 061 Hawke's Bay

081 Midcentral 082 Whanganui 091 Capital & Coast 092 Hutt 093 Wairarapa 101 Nelson Marlborough 111 West Coast 121 Canterbury 123 South Canterbury 160 Southern

Appendix 4: Correspondence with patients

Email correspondence From: Health Experience <feedback@myexperience.health.nz> To: patients supplied email address Subject: Feedback on your health care experience

[Practice name or logo]

Dear Mary Smith

We invite you to complete this survey about your experience of your own health care over the past 12 months. Taking part is voluntary but we would appreciate your feedback.

Click here to begin

The time it takes to complete the survey depends on your answers. Generally it takes 10 to 15 minutes, but may take longer if you have a lot you’d like to say. If you need some help to complete this survey, please ask a relative or friend. You have three weeks from the date of this email to respond before the survey is closed.

We take your privacy very seriously. Your response is anonymous. Unless you choose to provide your contact details in this survey, no one will be able see your name or any other contact details.

The feedback you give here is a very valuable way for us to understand how we can improve our service, so thank you in advance for your participation.

Yours sincerely

[Practice signatory] [Role title] [Practice name] [email] [phone number]

If you see your GP or Nurse more than three months from now you may receive this survey again. Please unsubscribe from this list if you do not wish to receive another email.

Reminder email Subject: Reminder - Feedback on your health care experience

[Practice name or logo]

Dear Mary Smith

Recently we sent you an invitation to complete a survey about your experience of your own health care over the past 12 months. Taking part is voluntary, but we would appreciate your feedback.

Click here to begin

The time it takes to complete the survey depends on your answers. Generally it takes 10 to 15 minutes, but may take longer if you have a lot you’d like to say. If you need some help to complete this survey, please ask a relative or friend. You have two weeks from the date of this email to respond before the survey is closed.

We take your privacy very seriously. Your response is anonymous. Unless you choose to provide your contact details in this survey, no one will be able see your name or any other contact details.

The feedback you give here is a very valuable way for us to understand how we can improve our service, so thank you in advance for your participation.

Yours sincerely

[Practice signatory] [Role title] [Practice name] [email] [phone number]

If you see your GP or Nurse more than three months from now you may receive this survey again. Please unsubscribe from this list if you do not wish to receive another email.

SMS correspondence Note that SMS correspondence is constrained by the maximum number of characters per SMS (160) and associated costs. Both the initial and reminder SMS are the same.

Please give us feedback on your health care.Go to www.myexperience.health.nz/?s=XXXXXXXX Taking part is voluntary & anonymous.Thanks [Practice Name – 27 characters allowed]

Survey introduction Once respondents have clicked on the link to complete the online survey, they are taken to the following introduction page.

[Practice name or logo]

Thank you for taking the time to complete this survey.

This survey is about your experience of your own health care over the past 12 months. Your response is anonymous. Unless you choose to provide your contact details in this survey, no one will be able see your name or any other contact details, so please be open and frank in your feedback.

There are several places in this survey where you can explain things in your own words. It is important not to mention any names or specific details if you wish to remain anonymous.

The time it takes to complete the survey depends on your answers. Generally it takes 10 to 15 minutes, but may take longer if you have a lot you’d like to say. If you need some help to complete this survey, please ask a relative or friend. The survey closes three weeks from the date you received the email or text invitation.

Warm regards

[Practice contact details]

Click here to begin

Survey conclusion

Thank you for your time and feedback. You have now finished this survey.

We have recorded all your answers so you can now close this window.

Thanks again,

[Practice Signatory]

Appendix 5: Bias in response

The purpose of looking at results between different sub-groups of responders is to consider whether there are systematic patterns of different responses between different groups. This is important for the purposes of weighting responses. (Over-representation of a group which is likely to respond to questions differently will skew the overall result.)

Typically age, gender, method of admission, method of response and more controversially ethnicity are included as variables on which surveys are weighted. Each is considered in turn below.

Age – older people are happier

15-24 25-44 45-64 65-74 75-84 85+ Communication Q1 Qs answered 6.90 8.17 8.57 9.33 9.25 9.17

Q2 Explanations 7.75 8.49 8.92 8.95 8.97 10.00 Q3a Dr listened 8.25 8.73 8.92 9.12 9.50 8.57 Q3b Nurse

listened 8.33 8.67 8.77 8.77 8.38 9.29

Q3c Other listened

8.42 8.97 8.81 8.27 8.82 9.17

Q4 Side effects explained

5.71 5.20 5.87 5.81 5.17 6.25

Participation Q6 Involved 8.33 8.15 7.73 8.33 8.82 9.29 Q7 Family

involved 7.75 7.32 7.70 6.78 7.78 9.29

Coordination Q9 Different things said

6.90 7.88 8.23 8.95 8.75 9.29

Q10 Discharge info

6.50 5.98 7.66 7.73 8.11 7.50

Physical and emotional needs

Q12 Toilet 7.50 8.19 8.45 8.92 8.97 8.75 Q13 Pain 8.33 8.83 9.10 8.90 9.29 10.00 Q14 Respect 8.33 9.09 9.38 9.04 9.63 10.00 Q15 Kindness 8.33 8.71 9.08 9.21 9.75 10.00 Q16 Culture 8.57 8.08 8.82 9.55 10.00 -

Overall Q20a Confidence Dr

7.75 8.73 9.00 9.39 9.75 10.00

Q20b Confidence Nurse

7.75 8.52 9.22 9.04 9.38 9.29

Q20c Confidence Other

7.89 8.44 8.79 9.12 9.00 9.17

Questions with Sig Low Aggregate Scores

1 1 0 0 0 0

Questions with Average Aggregate Scores

17 17 18 15 13 11

Questions with Sig High Aggregate Scores

0 0 0 3 5 6

Gender – Men are happier

Female Male Communication Q1 Qs answered 8.27 9.16

Q2 Explanations 8.45 9.27 Q3a Dr listened 8.71 9.33 Q3b Nurse

listened 8.34 9.18

Q3c Other listened 8.70 8.74 Q4 Side effects

explained 4.69 6.84

Participation Q6 Involved 7.98 8.66 Q7 Family

involved 7.27 7.80

Coordination Q9 Different things said

7.83 9.07

Q10 Discharge info

6.55 8.28

Physical and emotional needs

Q12 Toilet 8.27 9.00 Q13 Pain 8.92 9.05 Q14 Respect 8.90 9.69 Q15 Kindness 8.74 9.64 Q16 Culture 8.55 9.25

Overall Q20a Confidence Dr

8.81 9.48

Q20b Confidence Nurse

8.56 9.48

Q20c Confidence Other

8.60 8.97

Questions with Sig Low Aggregate Scores 0 0 Questions with Average Aggregate Scores 18 18 Questions with Sig High Aggregate Scores 0 0

(note: none of these reach α=0.95 but several reach α=0.9, I anticipate that this will change as we get more data)

Ethnicity - no obvious variation between European and Māori responses (not enough data for other ethnic groups)

European Māori Communication Q1 Qs answered 8.75 8.24

Q2 Explanations 8.75 8.61 Q3a Dr listened 8.92 9.44 Q3b Nurse listened 8.72 8.61 Q3c Other listened 8.82 8.24 Q4 Side effects

explained 5.54 7.69

Participation Q6 Involved 8.24 7.50 Q7 Family

involved 7.57 6.54

Coordination Q9 Different things said

8.30 8.06

Q10 Discharge info 7.14 8.24 Physical and emotional needs

Q12 Toilet 8.67 7.50 Q13 Pain 9.03 9.12 Q14 Respect 9.32 8.61 Q15 Kindness 9.17 8.89 Q16 Culture 9.11 8.75

Overall Q20a Confidence Dr 9.19 9.44 Q20b Confidence

Nurse 9.07 8.89

Q20c Confidence Other

8.91 7.94

Questions with Sig Low Aggregate Scores Low 0 0 Questions with Average Aggregate Scores Ave 18 18 Questions with Sig High Aggregate Scores High 0 0

Admission method - no obvious variation between methods

Arranged Acute Waiting list

Communication Q1 Qs answered 7.76 8.79 8.89 Q2 Explanations 8.75 8.57 9.31 Q3a Dr listened 8.75 8.97 9.38 Q3b Nurse listened 8.50 8.64 8.92 Q3c Other listened 8.28 8.94 8.64 Q4 Side effects

explained 5.53 5.45 5.96

Participation Q6 Involved 8.50 7.99 8.67 Q7 Family involved 7.22 7.64 7.55

Coordination Q9 Different things said

7.83 8.40 8.46

Q10 Discharge info 7.41 7.01 7.85 Physical and emotional needs

Q12 Toilet 9.29 8.71 8.30 Q13 Pain 8.33 9.25 8.83 Q14 Respect 8.83 9.38 9.15 Q15 Kindness 8.50 9.22 9.15 Q16 Culture 7.78 9.05 9.44 Q20a Confidence Dr 8.57 9.11 9.54 Q20b Confidence

Nurse 8.62 8.94 9.22

Q20c Confidence Other

8.93 8.62 9.17

Questions with Sig Low Aggregate Scores 0 0 0 Questions with Average Aggregate Scores 18 18 18 Questions with Sig High Aggregate Scores 0 0 0

Distribution method – email and post seem to have some more positive responses (not clear if there is an age effect going on here)

cell email post Communication Q1 Qs answered 7.88 9.10 9.66

Q2 Explanations 8.27 9.31 8.67 Q3a Dr listened 8.50 9.45 8.83 Q3b Nurse

listened 8.51 8.77 8.83

Q3c Other listened 8.85 8.50 9.04 Q4 Side effects

explained 4.94 6.01 6.43

Participation Q6 Involved 7.70 8.58 9.11 Q7 Family

involved 7.32 7.39 8.40

Coordination Q9 Different things said

7.63 8.86 8.83

Q10 Discharge info

6.47 7.64 8.57

Physical and emotional needs

Q12 Toilet 8.26 8.62 9.38 Q13 Pain 8.81 9.14 8.93 Q14 Respect 9.09 9.27 9.33 Q15 Kindness 8.75 9.27 9.67 Q16 Culture 8.20 9.29 10.00 Q20a Confidence

Dr 8.50 9.59 9.33

Q20b Confidence

Nurse 8.50 9.27 9.14

Q20c Confidence

Other 8.38 8.92 9.42

Questions with Sig Low Aggregate Scores 0 0 0 Questions with Average Aggregate Scores 18 15 16 Questions with Sig High Aggregate Scores 0 3 2

Appendix 6: Survey data file

COLUMN HEADING DEFINITION COMMENT

Q1

Could you tell us if you are answering this survey on behalf of yourself or someone else?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q1_QO2_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q2 Is there one GP clinic or community health clinic you usually go to?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q3

You are enrolled at [PracticeName]. Is there a reason you don’t usually go there?

Free text field question answers are exported with the actual verbatim given.

Q4

Does the place you usually go to have a phone service, doctor or nurse available during evenings, nights or weekends? (Either there or another place)

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q5 Is there one GP or nurse you usually see? As above

Q6 How long have you been going to your current GP or nurse clinic? As above

Q7_SP1

When you ring to make an appointment how quickly do you usually get to see...Your current GP? As above

Q7_SP2

When you ring to make an appointment how quickly do you usually get to see…Any other GP at the clinic you usually go to? As above

Q7_SP3 When you ring to make an appointment how quickly do you usually get to As above

see…A nurse at the clinic you usually go to?

Q8 Is this acceptable? As above

Q8_QO2_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q9_SP1

How long do you usually have to wait for your consultation to begin with…Your current GP?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q9_SP2

How long do you usually have to wait for your consultation to begin with…Any other GP at the clinic you usually go to? As above

Q9_SP3

How long do you usually have to wait for your consultation to begin with…A nurse at the clinic you usually go to? As above

Q10 Is this acceptable? As above

Q10_QO2_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q11 Did the reception and admin staff treat you with respect?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q12

In the last 12 months was there a time when you did not visit a GP or nurse because of cost? As above

Q13 Could you tell us why cost stopped you from seeing a GP or nurse?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell.

Q13_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q14

Was there ever a time when you wanted health care from a GP or nurse but you couldn’t get it?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q14_QO2_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q15

When you contact your usual GP clinic about something important, do you get an answer the same day?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q16 Does your GP or nurse explain things in a way that is easy to understand? As above

Q17 Are you confident that your GP or nurse is aware of your medical history? As above

Q18

Have you been involved in decisions about your care and treatment as much as you wanted to be? As above

Q19_SP1 Does your GP or nurse...Treat you with respect? As above

Q19_SP2 Does your GP or nurse...Treat you with kindness and understanding? As above

Q19_SP3 Does your GP or nurse…Listen to what you have to say? As above

Q19_SP4 Does your GP or nurse…Spend enough time with you? As above

Q20

Is there anything you would like to tell us about your experience with your GP or nurse?

Free text field question answers are exported with the actual verbatim given.

Q21 Overall, was your experience with your GP or nurse clinic? 0 - 10 rating given by the patient.

Q22

Do you take any medication regularly? This includes vitamins, pain killers, supplements and any prescribed medication.

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q23_SP1

In the last 12 months...Did you have any tests such as x-rays, scans, blood tests or other tests? As above

Q23_SP2

In the last 12 months...Have you seen any health care professionals other than a GP or nurse? Some examples are a midwife, physiotherapist, psychologist, social worker, counsellor, pharmacist, or dietitian. As above

Q23_SP3 In the last 12 months...Have you seen a specialist doctor, other than a GP? As above

Q23_SP4

In the last 12 months...Have you been to the emergency department at the public hospital? As above

Q23_SP5 In the last 12 months...Have you stayed in hospital overnight? As above

Q23_SP6

In the last 12 months...Do you have a health condition that will last more than 6 months? As above

Q24

Were you involved as much as you wanted to be in decisions about the best medication for you? As above

Q25_SP1 Was the purpose of the medication properly explained to you? As above

Q25_SP2

Were the possible side effects of the medication explained in a way you could understand? As above

Q25_SP3

Were you told what could happen if you didn’t take the medication, in a way you could understand? As above

Q25_SP4 Were you told what to do if you experienced side effects? As above

Q26 Did you follow the instructions when you took the medication? As above

Q27

You said that you did not always follow the instructions when you took the medication. Please tell us why.

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell.

Q27_OtherText Open comment field. Free text field question answers are exported with the actual verbatim given.

Q28 Has cost stopped you from picking up a prescription?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q29

In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist (outside of hospital)? As above

Q30_SP1 Because of the wrong medication or dose, did you…Stop taking it? As above

Q30_SP2 Because of the wrong medication or dose, did you…Get medical advice? As above

Q30_SP3 Because of the wrong medication or dose, did you…Get medical care? As above

Q30_SP4

Because of the wrong medication or dose, did you…Get admitted to hospital? As above

Q31

Is there anything else you’d like to tell us about being given the wrong medication or dose?

Free text field question answers are exported with the actual verbatim given.

Q32

Is there anything you would like to tell us about your experiences with your medication?

Free text field question answers are exported with the actual verbatim given.

Q33

Was the need for the x-ray, test or scan(s) explained in a way you could understand?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q34

Were you told how you could find out the results of your x-ray, scan(s) or tests? As above

Q35

Were the results of the x-ray, test or scan(s) explained in a way you could understand? As above

Q36

Is there anything you would like to tell us about your experiences with x-rays, scan(s) or tests?

Free text field question answers are exported with the actual verbatim given.

Q37

Was there a time when test results or information was not available at the time of your appointment with the health care professional?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q38

Were you given conflicting information by different doctors or health care professionals e.g. one would you tell you one thing and then another would tell you something different? As above

Q39 How quickly do you usually get to see a health care professional? As above

Q40 Has cost stopped you from seeing a health care professional? As above

Q41 Could you tell us why cost stopped you from seeing a health care professional?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and seperated by a pipe within the one cell.

Q41_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q42

Is there anything you would like to tell us about your experiences with health care professionals (other than a doctor or nurse)?

Free text field question answers are exported with the actual verbatim given.

Q43

When you were referred to a specialist did you have any difficulties getting an appointment?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q44 Any comments? Free text field question answers are exported with the actual verbatim given.

Q45

In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q46 Any comments? Free text field question answers are exported with the actual verbatim given

Q47 Has cost stopped you from seeing a specialist doctor?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q48 Could you tell us why cost stopped you from seeing a specialist doctor?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell.

Q48_OtherText Open comment field Free text field question answers are exported with the actual verbatim given.

Q49_SP1

When you received care or treatment from specialist doctors, did they do the following? … Ask what is important to you?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q49_SP2

When you received care or treatment from specialist doctors, did they do the following? … Tell you about treatment choices in ways you could understand? As above

Q49_SP3

When you received care or treatment from specialist doctors, did they do the following? … Involve you in decisions As above

about your care or treatment as much as you wanted to be?

Q50 Any comments? Free text field question answers are exported with the actual verbatim given.

Q51

Do the specialist doctors know your medical history and the reason for your visit?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q52

Has a doctor ordered a test (e.g. blood test, x-ray etc) that you felt you didn’t need because the test had already been done? As above

Q53

Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? As above

Q54

Is there anything you would like us to know about how well your GP and specialist doctors are working together?

Free text field question answers are exported with the actual verbatim given.

Q55

The last time you went to the public hospital emergency department, why did you go there?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell.

Q55_OtherText Any comments? Free text field question answers are exported with the actual verbatim given

Q56

Is there anything you would like us to know about how well your GP and the emergency department are working together?

Free text field question answers are exported with the actual verbatim given.

Q57 Could you please tell us which hospital you received care from?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q58

Did the hospital arrange follow-up care with a doctor or other health care professional? As above

Q59 Did your current GP seem informed and up-to-date about the plan for follow-up? As above

Q60

Did you have to go back to hospital or get emergency care because of complications within a month after being discharged from hospital? As above

Q61

Is there anything you would like to tell us about your experience of your GP and the hospital working together?

Free text field question answers are exported with the actual verbatim given.

Q62_SP1 How long ago were you first diagnosed for the condition(s)? … Anxiety

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q62_SP2 How long ago were you first diagnosed for the condition(s)? … Arthritis As above

Q62_SP3 How long ago were you first diagnosed for the condition(s)? … Asthma As above

Q62_SP4 How long ago were you first diagnosed for the condition(s)? … Cancer As above

Q62_SP5

How long ago were you first diagnosed for the condition(s)? … Chronic Obstructive Pulmonary Disease (COPD) As above

Q62_SP6 How long ago were you first diagnosed for the condition(s)? … Depression As above

Q62_SP7 How long ago were you first diagnosed for the condition(s)? … Diabetes As above

Q62_SP8 How long ago were you first diagnosed for the condition(s)? … Heart disease As above

Q62_SP9

How long ago were you first diagnosed for the condition(s)? … High blood pressure As above

Q62_SP10 How long ago were you first diagnosed for the condition(s)? … Long term pain As above

Q62_SP11

How long ago were you first diagnosed for the condition(s)? … Other mental health conditions As above

Q62_SP12 How long ago were you first diagnosed for the condition(s)? … Stroke As above

Q62_SP13 How long ago were you first diagnosed for the condition(s)? … Other As above

Q63 Which other long-term condition(s) have you been diagnosed with?

Free text field question answers are exported with the actual verbatim given.

Q64_SP1

Were you given information you could understand about things you should do to improve your health?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q64_SP2

Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life? As above

Q64_SP3

After a treatment or care plan was made were you contacted to see how things were going? As above

Q65

Is there anything you would like to tell us about your experience being treated for a long term condition?

Free text field question answers are exported with the actual verbatim given.

Gender (respondent) Are you...

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Ethnicity Which ethnic group or groups do you belong to?

Ethnicity is sourced from the patient’s answer to the survey question(s) and mapped according to the Ethnicity protocol codeset. Note that multiple selections are allowed and separated by a pipe within the one cell.

Ethnicity Other

You selected 'other' as an option for your ethnic group. Which of these ethnic groups do you belong to? As above

Q70

In which language(s) could you have a conversation about a lot of everyday things?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell.

Q70_OtherText Any comments? Free text field question answers are exported with the actual verbatim given.

Q71

Did you need an interpreter to communicate with a health care professional?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q72 Any comments? Free text field question answers are exported with the actual verbatim given.

Q73

Did the interpreter help you clearly communicate with the health care professional?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Q74 Any comments? Free text field question answers are exported with the actual verbatim given.

Q75 Was cultural support available when you needed it?

Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

Age Group

Summarised into age groups from the year of birth the patient provides in the survey.

Gender Data is sourced from the patient extract file.

Ethnicity 1 Data is sourced from the patient extract file.

Ethnicity 2 Data is sourced from the patient extract file.

Ethnicity 3 Data is sourced from the patient extract file.

HPI-O (practice) HPI Facility ID for the practice. DHB of Domicile (patient) Practice DHB PHO DHB

Response ID Random unique ID automatically generated by the system.

StartedDate Date and time the survey was started.

LastActivityAt Date and time the survey was completed.

Appendix 7: Patient experience report examples

Dashboard view – the first page users land on is the dashboard. This gives an instant snapshot of key results, comparisons and ‘work-on’ areas.

Results menu

The left hand side menu closely matches the modules that patients are asked about in the survey. Each module is a section in the menu, all of which concertina open to reveal results about that area of care.

Trend reports - see trends over time with trend reporting on practices (subject to user access limitations), PHOs, DHBs or nationally.

Practice comparisons – practice level reports have limited access to protect patient anonymity and practice privacy. Compare the results of practices within your PHO.

Filters – report filters enable you to ‘place a lens’ on date ranges, age groups, gender and ethnicity.

Patient contact requests

Patients are given the opportunity to request contact from their practice in the survey. If they do so, a notification email will be sent to the nominated practice’s email address, along with an attachment of the patient’s full survey response (if they have given permission for their survey answers to be attributed to their contact details).

Appendix 8: Licensed software support

Cemplicity’s monthly fee includes:

a) Reasonable Support for PHOs through one nominated Administrator at each PHO by email and phone

b) Reasonable Support for the Commission’s nominated Administrator by email and phone c) Access to the reporting portal for 20 users at national organisations d) Access to the reporting portal for up to 2 users at each DHB e) Access to the reporting portal for 2 users at each PHO f) Access to the reporting portal for 1 user at each general practice g) Receipt of a data extract from each of the 6 PHOs and NES for the first two surveys

(interim solution) h) Receipt of a data extract from NES 4 times a year (once the ongoing NES solution is in

place) i) Sending survey invitations via email or SMS to patients under the 1,200 practice names

and signatories (if provided on the agreed time by PHOs) j) Presenting survey questions online to respondents under the 1,200 appropriate practice

names and signatories (if provided on the agreed time by PHOs) k) All email costs for the quarterly process and SMS costs to send 5,500 SMS messages

per quarter (if the message is under 160 characters) l) Access to all current and future features provided within General Releases m) Secure storage of non-identifiable patient data and survey responses n) The Domain, secure certificates and all data storage and hosting within New Zealand o) Access to an Action Register at each PHO on request at no additional cost.

The monthly fee does not include:

p) The cost of questions other than the Core Survey Tool q) Questions specific to any PHO or group of PHOs r) More frequent surveying than quarterly s) Delivery of the survey in any language other than English t) Data manipulation or cleansing if extracts do not meet specifications; (the upload

process has an in-built error-message for incorrectly formatted or empty fields) u) Any SMS costs other than those outlined above v) Additional analysis, reporting or presentations other than the online reporting w) Significant compliance activity that may be required over and above meeting the

requirements of the request for tender, specifically compliance activity at PHO level.