Predicting Patient ExperienceBy Richard Staelin, PhD

The value of patient experience data goes well beyond public reporting. Many healthcare professionals look to their patient experience data for clarity on how to improve service delivery and quality of care. As well they should, since the quality of healthcare depends in part on the exchange of information between patients and their healthcare providers. Patient feedback is a rich source of data for identifying points of satisfaction and areas for improvement.1

While there is a tendency to approach patient feedback as a record of what went well or what went wrong, there is much more that can be done with the insights provided by patients. Captured data can be used to anticipate the experiences of future patients so that healthcare professionals are equipped with information they need to manage and exceed patient expectations and actively reduce service failures.

What are the service factors and interactions that best predict when patients are delighted with the quality of service provided versus those that result in service failures from the perspectives of patients? We were interested in answering these questions. We also wanted to determine whether factors affecting patient satisfaction and dissatisfaction are consistent across service lines.

Captured data can be used to anticipate the experiences of future patients so that healthcare professionals are equipped with the information they need to manage and exceed patient expectations and actively reduce service failures.

BIVARUS.COM

Predicting Patient Experience

BIVARUS.COM

OUR APPROACH

To determine which variables would best predict the likelihood of patients to rate their experiences as superior (by offering a rating of 5 out of 5 on whether they would recommend the provider or facility) or indicate that they experienced service failures (by offering ratings of 1, 2 or 3 on whether they would recommend), we analyzed 13,820 patient survey responses collected for the dermatology center and 20,226 responses collected for the emergency department of a large academic healthcare system over the course of nine months.

Surveys were deployed and responses were captured using the Bivarus platform, which dynamically generates relevant, targeted survey measures for each patient. The survey measures targeted to specific patients are based on patient and service attributes, the relative importance of the measures to facility administrators and all prior responses captured by the system for a specific patient population. This approach allows providers and organizations to collect significant amounts of data on dozens of survey measures while patients are asked a maximum of 10 questions – dramatically increasing response rates. Specifically, patients were invited via email and text to provide information about their interactions with the organization’s care teams soon after their encounters. Patients received unique URLs to surveys containing questions specific to their care.

Our aim in this study was to analyze all survey responses to find sets of questions that best predict patients giving ratings of 5 or a 1, 2 or 3 on their likelihood to recommend the provider or facility. Once these question sets were identified, we examined which responses to specific questions were associated with overall high or low scores.

Because each patient was asked only 10 questions each (including a common question on likelihood to recommend), we were not able to run regression analyses. Instead, we organized analyses using decision trees in which branches looked for predictor variables that best separated all responses into either “good” or “bad” responses. The algorithm used for analyses was Extreme Gradient Boosting (XGBoost) derived from the gradient boosting concept proposed by Jerome Friedman in “Greedy Function Approximation: A Gradient Boosting Machine.”2 A single decision tree is usually not strong enough to be used in practice so the XGBoost algorithm uses an ensemble model, which sums up predictions of multiple decision trees together. Those results were aggregated to reveal variables found often enough be part of a predictive decision tree. For example, if the vast majority of males gave superior ratings and the vast majority of females didn’t, then this methodology might have the decision tree split on gender. The algorithm would then look at the next level to see which variable helps classify males (or females) into homogeneous groups where giving a superior rating (or not) is a characteristic of these groups.

Predicting Patient Experience

WHAT WE FOUND

There were some broad similarities between the data sets of the dermatology center and the emergency department. For example, patients who took advantage of open-text options in the surveys tended to report negative views of the services they received, highest ratings came from patients younger than 20 (in general these were patients where an accompanying adult was the respondent), lowest ratings came from people in their 20s with satisfaction rates increasing with age, and encounters or appointments that took place in the morning tended to receive higher ratings overall.

However, in looking at specific predictive variables of high and low ratings for each facility, there was quite a distinction. Patient responses for the dermatology center identified variables covering communication, overall length of appointment and doctor competence as major factors in the experiences of patients. And, it turns out that several specific communication variables are especially important for the dermatology center.

The likelihood of dermatology patients to recommend the center is directly affected by the success or failure of doctors or clinical staff to:

• Explain reasons for ordering tests.

• Explain diagnoses.

• Inform patients about treatment options and procedures.

• Make patients aware of delays.

Additionally, consistently engaging patients in their own care by taking their concerns seriously and involving them in treatment decisions will lead to a higher likelihood of patients recommending this dermatology center.

We also saw that variables above predicted either superior experiences or service failures – but not always both. For example, “explain reasons for ordering tests” was associated with service failures when patients rated this aspect of the service very low, but high ratings on this same variable did not predict a superior experience. Similarly, high ratings on “explain diagnoses” was only associated with service success – low ratings did not predict service failure. Communication variables are interpreted differently by patients. Some variables signal only superior service, others only service failures and still others signal both depending on the context and quality of the communication.

BIVARUS.COM

Predicting Patient Experience

BIVARUS.COM

Patient data for the emergency department indicated that care coordination, wait time, comfort and doctor competence are major factors in the experiences of patients. For this particular department, specific coordination and wait time variables proved to have the most direct impact.

Emergency department variables directly impacting patients’ likelihood to recommend the facility to friends and family:

• Total length of visit is reasonable.

• Wait time between seeing doctors is reasonable.

• Time in treatment room is reasonable.

• Time it takes to receive medication is reasonable.

• Good coordination at shift change.

• Doctors and nurses inform patients of how long things will take.

• Nurses answers patient questions in a timely manner.

• Easing patient anxiety is a priority.

• Treatment area is pleasant.

As with the dermatology center, some specific variables were associated only with perceptions of excellent or poor service. Low scores on the variables “doctors seemed rushed,” “my pain was treated effectively” and “my physical discomfort was taken seriously” were precursors of patients indicating service failure but high scores on these variables didn’t signal excellent service.

FINAL THOUGHTS

While providers and healthcare organizations benefit from the feedback of delighted patients, effectively addressing service failures and areas of dissatisfaction may improve outcomes for entire patient populations. Service failures are easier to attack or resolve when the individual factors leading to dissatisfaction among patients can be clearly identified. Our analyses demonstrate that service and quality improvements cannot be made using generalizations. The variables affecting the experiences of patients are entirely unique to different healthcare departments, services and providers. And, among those unique variables, some have more influence on perceptions of either superior service or service failure. By utilizing assessments tailored to their organization’s particular services, healthcare providers and leaders can effectively prioritize the types of improvements that will most directly impact the experiences of their patients.

Predicting Patient Experience

ABOUT RICHARD STAELIN

Rick serves as a scientific advisor to Bivarus and is the Edward and Rose Donnell Professor of Business Administration at The Fuqua School of Business, Duke University. Rick has published over 80 articles in academic journals on a diverse set of subjects including information search, managerial decision making, influence of verbal and pictorial messages in print ads, consumer protection regulation and its impact on consumer behavior, channel management strategic alliances, healthcare quality, and marketing strategy. He has taught in a number of executive programs around the world; his main interest in executive education is showing how strategic or tactical marketing problems can be better solved using good analysis.

Rick holds two bachelor degrees in Engineering, an MBA and a PhD from the University of Michigan.

ABOUT BIVARUS

Bivarus was founded by two doctors in response to the frustration clinicians share about the quality and usefulness of data about the patient experience. Supported by a cloud-based analytics platform that captures and delivers patient feedback in real time, we support healthcare leaders in moving the conversation from how to collect data to how to respond to that data. Healthcare organizations rely on us for:

• Statistically significant, actionable patient-experience data that can be operationalized toimprove service delivery.

• Intervention monitoring facilitated through real-time tracking and comprehensive reporting.

• Research around the patient experience and continuous improvements in approachesdesigned to measure, intervene in and monitor the patient experience.

For more information on how we can support your organization's efforts to measure and improve the experiences provided to patients, please visit bivarus.com, email info@bivarus.com or call us at (919) 336-9142.

BIVARUS.COM

Predicting Patient Experience

BIVARUS.COM

REFERENCES

1 PBoulding, William, Glickman, Seth W., Manary, Matthew P., & Staelin, Richard.

“The Patient Experience and Health Outcomes.” N Engl J Med 2013; 368:201-203.

2 Friedman, Jerome H. "Greedy Function Approximation: A Gradient Boosting Machine."

IMS 1999 Reitz Lecture.

http://www.nejm.org/doi/full/10.1056/NEJMp1211775#t=article

https://statweb.stanford.edu/~jhf/ftp/trebst.pdf