Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
1
1B - Why Measure?
Advanced Measurement for Improvement Seminar
March 26-27, 2015
Health Care
Produces these
outcomes
“The Gap”
Intention
to
improve
A Vision
A Plan
to
improve
Changes
Patients
Patients/Care seekers want these
outcomes
Advanced
Measurement
Seminar
Improvement Capability
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
2
Pre-work
Introduced three faces of performance
measurement:
�Research
�Accountability/Judgment
�Improvement
P3
Prework_Three_Types_of_Measurement.pptx
Research,Accountability,Improvement
Three faces of measurement
Translational research
Cooking the numbers
Why? Who?
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
3
3 Faces of Performance Measurement
Improvement Accountability Research
Aim Improvement of care Comparison, choice, reassurance, spur for change
New knowledge
Test Observability Test observable No test, evaluate current performance
Test blinded or controlled
Bias Accept consistent bias Measure and adjust to reduce bias
Design to eliminate bias
What to Measure “Just enough” data, small sequential samples
Obtain 100% of available, relevant data
“Just in case” data
Flexibility of
Hypothesis
Hypothesis flexible, changes as learning takes place
No hypothesis Fixed hypothesis
Testing Strategy Sequential tests No tests One large test
Determining if a
Change is an
Improvement
Run charts or Shewhart control charts
No change focus Hypothesis, statistical tests (t-test, F-test, chi square), p-
vlaues
Response to Context Variation
Utilize to test resilience of process design
Ignore, acknowledge, or use risk adjustment
Control or eliminate effects of confounding vbls
Confidentiality of
the Data
Data used only by those involved with improvement
Data available for public consumption and review
Research subjects’ identities protected
Solberg, L. I., G. Mosser, et al. (1997)
Translational Research
Westfall, J. M., J. Mold, et al. (2007), Khoury, M. J., M. Gwinn, et al. (2007).
Quality Improvement is a T3
enterprise
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
4
Measurement for Improvement
What's the logic of Measurement for Improvement?
(1) "If we measure, we have to improve."
Or
(2) "If we improve, we have to measure."
?
P8
Source: Kevin Little
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
5
Three ways to get better numbers P9
1. Improve the system.
2. Distort the system.
3. Distort the numbers.
Brian Joiner, Ph.D., statistician,
founder of Joiner Associates,
Kevin’s boss, teacher and friend.
He sketched his "better numbers"
analysis 30 years ago.
Source: Kevin Little
Campbell's Insight on Distortion
“The more any quantitative social
indicator is used for social decision-
making, the more subject it will be to
corruption pressures and the more apt
it will be to distort and corrupt the
social processes it is intended to
monitor.”
"Campbell's Law" from Assessing the Impact
of Planned Social Change, 1976
P10
Donald T. Campbell, Ph.D.
social psychologist
(1916-1996)
Source: Kevin Little
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
6
Distorting the system & the numbers
Measurement Use: Judge
schools and students as
successful based on the
scores from a competency test.
"Unless fifty-eight per cent of
students passed the math
portion of the test and sixty-
seven per cent passed in
language arts, the state could
shut down the school."
P11
www.newyorker.com/magazine/2014/0
7/21/wrong-answer
Source: Kevin Little
CMS Five-Star Nursing Home Ratings: Measurement Challenges
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
7
P13
http://www.medicare.gov/NursingHomeCompare/About/Ratings.html accessed 19 Nov 2014
Self-reported by
Nursing Homes
before 2015
Source: Kevin Little
P14
"Receiving a high star rating has never been more important to nursing homes.
When nurses and doctors discharge patients from hospitals, they often use the
ratings in referral decisions, and insurers consider them when setting up preferred
networks. The ratings are also often a first stop for investors and lenders, who
consult them to decide whether a nursing home company is a safe bet."
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
8
Visual Story: Text, numbers and repeated stacked bar graphs, over time
P15
http://www.nytimes.com/2014/10/07/business/medicare-alters-its-
nursing-home-rating-system.html
More Background-1
Up through 2014, staff ratings are derived from an annual self-report, completed at same time as the annual inspection.
Raising staff levels for the month when the annual inspection occurs can yield a higher score on the staff component and so boost the star rating.
CMS announced in October 2014 that starting in 2015 they will change the staffing component to a quarterly report, linked to electronic payroll data, potentially auditable.
(based on New York Times articles)
P16
Source: Kevin Little
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
9
More Background-2 P17
"Advocates for patients and for
large employers that pay for
healthcare services argue that the
government should not slow the
rollout. They acknowledge that
performance ratings are an
imperfect science, but counter that
more transparency and better tools
are needed to help consumers
make informed choices."
"Dr. Patrick Conway, deputy
administrator for innovation and
quality for the CMS, said in a June
blog post that ratings empower
consumers and encourage
providers to strive for higher levels
of quality."
http://www.modernhealthcare.com/article/20140830/NEW
S/308299954 accessed 19 Nov 2014
Source: Kevin Little
More Background-3 P18
Risk-
adjusted
http://www.cms.gov/Medicare/Provider-Enrollment-and-
Certification/CertificationandComplianc/Downloads/usersguide.pdf p. 10 accessed 19 Nov 2014
Advanced Measurement for Improvement
Cambridge, MA • March 26-27, 2015
10
Exercise: CMS Five-Star Ratings
CMS describes the Five-Star ratings this way:
“The primary goal of this rating system is to provide residents and their families with an easy way to understand assessment of nursing home quality, making meaningful distinctions between high and low performing nursing homes.”
(Design for Nursing Home Compare Five-Star Quality Rating System: Technical Users’ Guide, July 2012, p. 1)
1. Our view: the CMS Five-Star rating is a measurement system primarily for judgment/accountability.
Do you agree or disagree? Why?
2. Many people argue that transparent reporting of measures can drive better performance, reflecting improvements in nursing homes.
How could the Five-Star rating system be amended so that people will improve nursing homes rather than distort nursing homes or distort the reported numbers?
Discuss in pairs & share!
P19
Source: Kevin Little
Insights? Questions?
Top Related