The 'Tripadvisor' Approach to Health
-
Upload
harvardhighq -
Category
Health & Medicine
-
view
331 -
download
1
description
Transcript of The 'Tripadvisor' Approach to Health
![Page 1: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/1.jpg)
The Tripadvisor approach for health? Using patients' online descriptions of their care to understand healthcare quality
Felix Greaves
Imperial College London
@felixgreaves
![Page 2: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/2.jpg)
England is proud of the National Health Service
![Page 3: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/3.jpg)
But it is not without problems Quality can be variable, and we often fail to hear the patient perspective
![Page 4: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/4.jpg)
At the same time, we love rating things on the internet
![Page 5: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/5.jpg)
We now do it for our experiences of healthcare too
![Page 6: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/6.jpg)
The number of patients describing their care online is increasing
USA UK
Gao et al., JMIR, 2012 Greaves et al., JMIR, 2012
Aug-0
8
Nov-0
8
Feb-0
9
May
-09
Aug-0
9
Nov-0
9
Feb-1
0
May
-10
Aug-1
0
Nov-1
0
Feb-1
1
May
-11
Aug-1
1
Nov-1
10
5000
10000
15000
20000
25000
Nu
mb
er
of h
osp
ital r
atin
gs
![Page 7: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/7.jpg)
There is resistance from many clinicians to the idea of being rated online:
![Page 8: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/8.jpg)
Arguments for and against online rating
Why it’s bad
1. Strain doctor-patient relationships
2. Reviews may be malicious or fake
3. Selection bias by those leaving reviews
4. Lack of meaningful data on technical quality of health care
Why it’s good
1.Doctors can often be poor judges of their patients’ experience
2.Feedback changes doctors’ performance
3.People are using the Internet to voice opinions, so why not capture this information in a useful form
![Page 9: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/9.jpg)
Research Question
Is there a relationship between ratings online and ‘hard’ measures of quality?
![Page 10: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/10.jpg)
The opportunity for a natural experiment
NHS Choices website allows patients to review all NHS services in England
![Page 11: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/11.jpg)
Method:
Obtained national review data from the Department of Health
![Page 12: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/12.jpg)
We thought we could compare ratings with some traditional measures of quality
Vs.
![Page 13: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/13.jpg)
Average scores for 10,000 ratings
Score Mean rating (out of 5) Range
The environment where I was treated was…
3.6 2.6-5.0
I was treated with dignity and respect by the hospital staff…
4.0 2.7-5.0
I was involved with decisions about my care…
3.8 2.4-5.0
The hospital staff worked well together…
4.1 2.9-5.0
Greaves et al. BMJ Quality and Safety, 2012
![Page 14: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/14.jpg)
Average scores for 10,000 ratings
Score Mean rating (out of 5) Range
The environment where I was treated was…
3.6 2.6-5.0
I was treated with dignity and respect by the hospital staff…
4.0 2.7-5.0
I was involved with decisions about my care…
3.8 2.4-5.0
The hospital staff worked well together…
4.1 2.9-5.0
Greaves et al. BMJ Quality and Safety, 2012
• The majority of ratings are positive• 67% would recommend to a friend
![Page 15: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/15.jpg)
Ratings compared to patient experience surveys
NHS Choices Measure Survey question: Spearman
Rho p value
Proportion of patients recommending
“Overall, how would you rate the quality of care you received”
0.40 <0.001
Rating of being treated with dignity and respect
“Overall, did you feel you were treated with dignity and respect while in hospital?”
0.33 <0.001
Rating of staff working together
“How well would rate how well the doctors and nurse worked together?”
0.32 <0.001
Rating of cleanliness “How clean was the hospital ward or room you were in?”
0.48 <0.001
Greaves et al. BMJ Quality and Safety, 2012
![Page 16: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/16.jpg)
Ratings compared to patient experience surveys
NHS Choices Measure Survey question: Spearman
Rho p value
Proportion of patients recommending
“Overall, how would you rate the quality of care you received”
0.40 <0.001
Rating of being treated with dignity and respect
“Overall, did you feel you were treated with dignity and respect while in hospital?”
0.33 <0.001
Rating of staff working together
“How well would rate how well the doctors and nurse worked together?”
0.32 <0.001
Rating of cleanliness “How clean was the hospital ward or room you were in?”
0.48 <0.001
There is a moderate, highly significant association between ratings online and large surveys of patient experience
Greaves et al. BMJ Quality and Safety, 2012
![Page 17: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/17.jpg)
Ratings compared to outcomes
NHS Choices Measure Other variable Spearman Rho p value
Proportion of patients recommending
Hospital Standardised Mortality Ratio
-0.20 0.01
Proportion of patients recommending
Standardised morality rate for high risk conditions
-0.22 0.01
Proportion of patients recommending
Standardised morality rate among surgical inpatients with serious treatable complications
-0.00 0.99
Proportion of patients recommending
Emergency readmission rate within 28 days
-0.31 <0.001
Patient perception of cleanliness
Rate of MRSA bacteraemia (per 1,000 bed days)
-0.30 <0.001
Greaves et al. Arch Int Med, 2012
![Page 18: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/18.jpg)
Ratings compared to outcomes
NHS Choices Measure Other variable Spearman Rho p value
Proportion of patients recommending
Hospital Standardised Mortality Ratio
-0.20 0.01
Proportion of patients recommending
Standardised morality rate for high risk conditions
-0.22 0.01
Proportion of patients recommending
Standardised morality rate among surgical inpatients with serious treatable complications
-0.00 0.99
Proportion of patients recommending
Emergency readmission rate within 28 days
-0.31 <0.001
Patient perception of cleanliness
Rate of MRSA bacteraemia (per 1,000 bed days)
-0.30 <0.001
There is a weak, significant association between ratings online and mortality rates
Greaves et al. Arch Int Med, 2012
![Page 19: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/19.jpg)
A study in the US found the same results comparing Yelp reviews with the HCAHPS survey
![Page 20: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/20.jpg)
A wisdom in the crowd of patients?
Sir Francis Galton won ‘guess-the-weight-of-the-bull’ competitions by asking lots of local farmers for their best guess, and then taking the average – he gave us the idea of a ‘wisdom of crowds’
![Page 21: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/21.jpg)
Comparison of the NHS Inpatient Survey and ratings on the NHS Choices website
NHS Inpatient Survey NHS Choices ratings
Mechanism Paper-based survey Ratings left on a website
Number of responses
69,000 per year 5,000 per year
Selection Random; patients receive a survey requesting completion after leaving hospital
Self-selecting; patients are not solicited
Proportion positive
79% rated their overall care as excellent or very good
67% would recommend to a friend
Cost Likely more expensive Likely less expensive
![Page 22: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/22.jpg)
Comparison of the NHS Inpatient Survey and ratings on the NHS Choices website
NHS Inpatient Survey NHS Choices ratings
Mechanism Paper-based survey Ratings left on a website
Number of responses
69,000 per year 5,000 per year
Selection Random; patients receive a survey requesting completion after leaving hospital
Self-selecting; patients are not solicited
Proportion positive
79% rated their overall care as excellent or very good
67% would recommend to a friend
Cost Likely more expensive Likely less expensive
There were 10,000 hospital ratings in the UK over 2 years
Over the same time period there were 29,118,009 hospital admissions
0.04% of hospital admissions are rated
![Page 23: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/23.jpg)
The stereotypical reviewer?
![Page 24: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/24.jpg)
Where are people rating their care?
![Page 25: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/25.jpg)
Associations between whether a practice is rated with population and practice characteristics
Independent variable
Z statistic p value
Practice population size
15.38 <0.001
IMD score of patients
-7.82 <0.001
Population density
6.72 <0.001
Singlehander -4.50 <0.001Proportion of population aged over 65 years
-3.88 <0.001
Proportion of population who are white
-1.58 0.11
Type of contract -0.71 0.48Training practice 0.35 0.73
![Page 26: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/26.jpg)
Associations between whether a practice is rated with population and practice characteristics
Independent variable
Z statistic p value
Practice population size
15.38 <0.001
IMD score of patients
-7.82 <0.001
Population density
6.72 <0.001
Singlehander -4.50 <0.001Proportion of population aged over 65 years
-3.88 <0.001
Proportion of population who are white
-1.58 0.11
Type of contract -0.71 0.48Training practice 0.35 0.73
• Practices serving younger people are more likely to be rated
• Practices serving less deprived people are more likely to be rated
• Practices in urban areas are more likely to be rated
![Page 27: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/27.jpg)
Cloud of patient experience ?
![Page 28: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/28.jpg)
A problem
Free Text
![Page 29: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/29.jpg)
A new trend: ‘Big Data’ and social media analytics
![Page 30: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/30.jpg)
Sentiment analysis
![Page 31: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/31.jpg)
Another natural experiment
We can compare patients’ free text descriptions of care with their own quantitative ratings
![Page 32: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/32.jpg)
What we did: Machine learning
• You need to teach an algorithm how to recognise particular words and phrases
• We used all comments and ratings from 3 years as a training set (13,802 comments)
We tried to predict patients ratings of their care from their comments in 2010
• Whether the patient would recommend the hospital or not• Whether the hospital was clean or not• Whether the patient was treated with dignity or not
Used open source Weka software
![Page 33: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/33.jpg)
Sentiment analysis can be tricky
![Page 34: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/34.jpg)
Simple punctuation contains little information
(
![Page 35: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/35.jpg)
But when paired with other punctuation, it’s very important
:(
![Page 36: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/36.jpg)
And small changes can have big effects on sentiment
:)
![Page 37: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/37.jpg)
Some words the algorithm thought were important
Overall Cleanliness Dignity• rude • dirty • rude• excellent • floor • told• hours • filthy • thank• pain • bed • friendly• communication • blood • attitude
![Page 38: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/38.jpg)
How good is it at predicting ratings?
Question we are predicting the answer to
Total number of comments
Prediction accuracy
Kappa P value
Would you recommend the hospital?
6412 88.7 0.75 <0.0001
How clean was the hospital room or ward that you were in?
6139 81.2 0.40 <0.0001
Overall, did you feel you were treated with respect and dignity while you were in the hospital?
6239 83.6 0.56 <0.0001
![Page 39: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/39.jpg)
How does this compare to patient surveys?
Patient Survey Question Machine learning prediction
Spearman rho Probability
Overall, how would you rate the care you received?
Whether the patient would recommend the hospital 0.46 P<0.001
Overall, did you feel you were treated with respect and dignity while you were in the hospital?
Whether the patient was treated with dignity 0.50 P<0.001
In your opinion, how clean was the hospital room or ward that you were in?
Standard of cleanliness 0.37 P<0.001
![Page 40: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/40.jpg)
Limitations
• Selection bias• Sarcasm / Irony• Culturally specific
Phrases like ‘cup of tea’ important,
but effect depends on context
![Page 41: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/41.jpg)
The UK quality regulator nowincludes online reviews and comments in its performance measurement framework
![Page 42: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/42.jpg)
The NHS has started publically reporting social media sentiment
![Page 43: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/43.jpg)
ConclusionsOnline rating and reviewing are on the rise
Evidence of some wisdom in the crowd of patients
Substantial professional resistance to the idea
Selection bias in those commenting online
The ‘cloud of patient experience’ is a potentially valuable source of information to understand patient views
![Page 44: The 'Tripadvisor' Approach to Health](https://reader035.fdocuments.in/reader035/viewer/2022062708/558bfbced8b42a253c8b4740/html5/thumbnails/44.jpg)
• Christopher Millett• Ara Darzi• Dominic King• Henry Lee• Utz Pape• Liam Donaldson• Azeem Majeed• Robert Wachter• Daniel Ramirez-Cano
Thanks to my co-authors, and funders