Ashish K. Jha, MD, MPH · Ashish Jha, MD, MPH Ashish K. Jha, MD, MPH Director, Harvard Global...

1

August 21, 2018

Georg Roeggla

Associate Editor, The BMJ

Dear Dr. Roeggla,

Thank you for giving us an opportunity to revise our manuscript “Patient Outcomes and Accreditation in

U.S. Hospitals: An Observational Study.” We appreciate the issues raised by the reviewer and the editor’s

requests, and we are grateful for the opportunity to respond. Additionally, since the original submission of

our manuscript, we have obtained 2015 Medicare data and updated our results. Below are the comments

we received from the editor and reviewers (in italics) and our responses (in bold).

We believe that the reviewer comments were very helpful and addressing them has strengthened the

manuscript. We hope you agree. If there are any further issues or concerns that we can address, please do

not hesitate to contact us.

Sincerely,

Ashish Jha, MD, MPH

Ashish K. Jha, MD, MPH

Director, Harvard Global Health Institute

K.T. Li Professor of Global Health

Professor of Medicine

Harvard Medical School

2

14-Jun-2018

Dear Dr. Jha

Manuscript ID BMJ.2018.044844 entitled "Patient Outcomes and Accreditation in U.S. Hospitals: An

Observational Study"

Thank you for sending us your paper. We sent it for external peer review and discussed it at our

manuscript committee meeting. We recognize its potential importance and relevance to general medical

readers, but I am afraid that we have not yet been able to reach a final decision on it because several

important aspects of the work still need clarifying.

We hope very much that you will be willing and able to revise your paper as explained below in the

report from the manuscript meeting, so that we will be in a better position to understand your study and

decide whether the BMJ is the right journal for it. We are looking forward to reading the revised version

and, we hope, reaching a decision.

Please remember that the author list and order were finalized upon initial submission, and reviewers and

editors judged the paper in light of this information, particularly regarding any competing interests. If

authors are later added to a paper this process is subverted. In that case, we reserve the right to rescind

any previous decision or return the paper to the review process. Please also remember that we reserve

the right to require formation of an authorship group when there are a large number of authors.

Yours sincerely,

Georg Roeggla

[email protected]

**Report from The BMJ’s manuscript committee meeting**

These comments are an attempt to summarise the discussions at the manuscript meeting. They are not

an exact transcript.

Manuscript meeting 14.06.2018

Elizabeth Loder (chair), Angela Wade (stats), Wim Weber, John Fletcher, Georg Roggla, Tiago Villanueva.

Decision: Ask for revision

Thank you for the opportunity to address the questions and comments raised by the committee and

the reviewers.

3

The committee was interested in the topic of your research. The following concerns were mentioned.

• We think your paper is of interest not only to US readers. But we think it would be very helpful to

explain The Joint Commissions work in more detail. What is the process? What happens during

accreditation?

Thank you for this suggestion. We have now added additional information and references about the

Joint Commission as well as more details in general on the process of accreditation by other

accrediting organizations (AOs) and the state survey to the introduction. Specifically, we write the

following:

“The U.S. hospital accreditation process varies between state survey agencies and AOs.16–20

A

hospital that elects to undergo survey by a state agency can expect an annual, unannounced,

onsite inspection that determines their accreditation status. These reviews vary in length and

usually ensure that the hospital has people and policies needed to provide adequate quality

care. AOs are required to inspect hospitals at least every three years. TJC, for example,

performs unannounced onsite surveys for its clients every 18 to 36 months, whereas DNV GL,

a newer AO, performs annual onsite inspections. Additionally, AOs tend to provide more

structure, consulting with hospitals on how to prepare for an inspection and often have

additional quality metrics that they choose to examine. During the on-site inspection,

surveyors observe a broad range of hospital operations, but the focus is still largely on

structural factors and processes.19

There is little focus by AOs on whether the hospital is

achieving good outcomes.”

• Where are the joint commission accreditations performed apart from the US?

The international branch of the Joint Commission currently accredits over 1000 organizations in over

60 countries. The Joint Commission accredits many aspects of the health care system, from hospitals

and ambulatory surgical centers to nursing homes. We have added this information to the

introduction.

“Moreover, the international branch of TJC currently accredits over 1000 organizations in over

60 countries outside the U.S.7”

• Please explain the differences in the process when accreditation is performed by other

organisations.

Thank you – in response to the committee’s prior suggestion regarding explaining the Joint

Commission and the accreditation process, we added information regarding other AOs as well to the

introduction. Please see response above.

• The committee shares the concern that the information on accreditation status and mortality

and readmission were not collected within the same time period. Accreditation status is collected

between 2014 and 2017 and outcome data between January 1, 2014 and November 30, 2014. Please

explain how hospitals that undertake surveys in 2016 or 2017 are likely to perform better on patient

outcomes in 2014?

Thank you for this important comment, which has been echoed below by one of the reviewers.

4

We initially performed our mortality and readmissions analyses with the most contemporary

Medicare data we had available at the initiation of our project. We included all hospitals accredited

between 2014-2017 to analyze all U.S. hospitals. Hospitals have typically undergone accreditation for

years, if not decades, and therefore, we assumed that a single survey visit would not necessarily lead

to improved patient outcomes. As the Joint Commission states on their website, “accreditation is

about continuous quality — not ‘passing’ a survey”,1 and we expected that patients receiving care at

hospitals with accreditation would receive a standard quality of care irrespective of the most recent

inspection date. In fact, that’s consistent with the evidence that while there are transient changes in

hospital practice around the inspection date, the outcomes in a hospital does not vary based on when

the last inspection occurred.

It was with this empirical background that we focused on all hospitals that were accredited using the

most recent data we have and compared it to outcomes using the most recent data we had. We were

further assured of this approach given that it is very rare for hospitals to change accrediting

organizations.

Since our initial analyses, we have obtained 2015 Medicare data. We performed a sensitivity analysis

using this more recent data and our results are quite similar overall (Table 1, Table 2). There is no

significant difference in medical and surgical mortality for privately accredited hospitals versus those

undergoing a State Survey. There is no difference in surgical readmission rates and a trend toward

lower medical readmissions for privately-accredited hospitals, which is not significant based on our

pre-specified p-value. We have included the results of this sensitivity analysis in our manuscript

(Appendix Tables 6 and 7).

Finally, we performed one additional sensitivity analysis using the updated 2015 Medicare data on a

subset of hospitals, specifically those that were accredited in 2014 only (Table 3, Table 4). This most

directly addresses the issue of whether accreditation is associated with better outcomes in the

subsequent period.

This approached substantially decreased our hospital sample and the number of hospitalizations.

These results are largely similar with similar effect estimates and no statistically significant differences

based on our pre-specified p-values. There is a significantly higher surgical mortality rate for non-TJC

hospitals compared to TJC hospitals. Here, because we are working with a small sample size, it

appears that non-TJC privately accredited hospitals that happened to be accredited in 2014 had

particularly poor surgical outcomes that one year – a trend we don’t see when we examine

subsequent years of non-TJC accredited hospitals. We have included the results of this sensitivity

analysis in our manuscript (Appendix Tables 8 and 9).

Table 1. 30-Day Risk-Adjusteda Mortality and 30-Day Risk-Adjusted

a Readmission Rates for Hospitals

Accredited by an Accrediting Organization (AO) vs. Reviewed by a State Survey Agency (2015 Medicare

data)

AO

% [95% CI] State Survey

% [95% CI] Difference

% [95% CI] p-value

b

30-Day Mortality

Medical 10.3%

[10.2% - 10.4%] 10.6%

[10.3% - 10.9%] -0.3%

[-0.7% - 0.04%] 0.08

5

Surgical 2.14%

[2.09% - 2.18%] 2.07%

[1.8% - 2.4%] 0.07%

[-0.2% - 0.4%] 0.67

30-Day Readmissions

Medical 22.3%

[22.2% - 22.4%] 22.9%

[22.4% - 23.4%] -0.6%

[-1.1% - -0.1%] 0.02

Surgical 14.9%

[14.7% - 15.2%] 14.3%

[13.0% - 15.6%] 0.7%

[-0.6% - 2.0%] 0.30

aAdjusted for patient and hospital characteristics and HRR fixed effects. bA p-value less than 0.0125 was considered significant.



Accredited by TJC vs. non-TJC (2015 Medicare data)

TJC

% [95% CI] Non-TJC


% [95% CI] p-value

b

30-Day Mortality

Medical 10.3%

[10.2% - 10.4%] 10.2%

[10.0% - 10.5%] 0.04%

[-0.2% - 0.3%] 0.75

Surgical 2.15%

[2.10% - 2.20%] 2.2%

[2.0% - 2.3%] -0.03%

[-0.2% - 0.1%] 0.72

30-Day Readmissions

Medical 22.3%

[22.1% - 22.4%] 22.4%

[22.0% - 22.8%] -0.1%

[-0.5% - 0.3%] 0.51

Surgical 15.0%

[14.7% - 15.3%] 15.1%

[14.3% - 15.9%] -0.1%

[-0.9% - 0.7%] 0.82




Accredited by an Accrediting Organization (AO) vs. Reviewed by a State Survey Agency (2015 Medicare

data for 2014 accreditation)

AO

% [95% CI] State Survey


% [95% CI] p-value

b

30-Day Mortality

Medical 10.3%

[10.17-10.45%] 10.8%

[10.40-11.20%]

-0.49%

[-0.96- -0.02%] 0.04

Surgical 2.0%

[1.90-2.07%] 2.1%

[1.70-2.47%]

-0.10%

[-0.54-0.34%] 0.66

30-Day Readmissions

Medical 21.9%

[21.67-22.16%] 22.8%

[22.13-23.43%]

-0.87%

[-1.61- -0.13%] 0.02

Surgical 13.7%

[13.24-14.13%] 13.1%

[11.38-14.77%]

0.61%

[-1.27-2.49%] 0.52




Accredited by TJC vs. non-TJC (2015 Medicare data for 2014 accreditation)

TJC Non-TJC Difference p-value

b

6

% [95% CI] % [95% CI] % [95% CI]

30-Day Mortality

Medical 10.3%

[10.15-10.39%] 11.1%

[10.40-11.74%]

-0.80%

[-1.50- -0.10%] 0.03

Surgical 2.1%

[1.99-2.12%] 2.8%

[2.36-3.14%]

-0.70%

[-1.11- -0.28%] 0.001

30-Day Readmissions

Medical 21.9%

[21.66-22.11%] 22.8%

[21.50-24.17%]

-0.96%

[-2.33-0.42%] 0.17

Surgical 13.9%

[13.48-14.28%] 15.0%

[13.16-16.86%]

-1.13%

[-3.08-0.81%] 0.25


• Please discuss pros and cons of the accreditation process. Is marketing a major issue?

Due to the increased focus on healthcare quality and patient safety, there has been greater

expectation from hospitals and hospital leadership on providing evidence of their commitment. This

in turn has led to the use of hospital accreditation as a way to market high quality healthcare.

While this has been part of the U.S. system for many decades, use of hospital accreditation is now

starting to take off globally. The assumption is that if hospitals pay large sums of money to

accreditors, they will get the “gold seal of approval” which can assure consumers quality and lead to

substantial revenues for hospitals. Our findings call into question to what degree this gold seal of

approval really means better outcomes.

There is no doubt accreditation has value. It forces hospitals to examine their staffing, their

processes, and to pay some attention to quality. There is some evidence that accreditation leads to a

bit higher rates of adherence to guidelines that are being tracked.

However, the cons are that the process is expensive, cumbersome, and some clinicians feel it takes

away from their focus on clinical care. And of course, it is a major source of marketing by the

hospitals and an excellent business model for the accrediting agencies – which also sell consulting

services to help hospitals get ready for their inspectors and consulting services which help hospitals

address problems found by the inspectors.

• We liked that you look at patient outcomes and not just processes of care.

Thank you for your comment. We agree that it is important to look at all aspects of care, especially

patient outcomes.

• Could you give some more information, perhaps in a box or elsewhere, about the differences

between hospitals that are and are not JC accredited, e.g. size, geographic location, patient mix, etc.

7

Thank you for this comment. We agree that there are baseline differences in hospitals that are

accredited. Table 1 highlights some of the differences in patient characteristics (age, race, sex,

Medicaid eligibility, and comorbidities) and hospital characteristics (size, ownership, teaching status,

geographic location, urban location, critical access hospital, and ICU). We have added additional

information to the results section to further describe these and a new table in the appendix.

“Compared to hospitals accredited by TJC or non-TJC organizations, hospitals reviewed by

State Survey were more often smaller, non-teaching institutions, more likely to be located in

rural areas and lacking an intensive care unit (ICU). TJC hospitals are more likely to be larger,

teaching institutions, located in urban locations, and in the northeast and south as compared

to hospitals accredited by non-TJC organizations or reviewed by a State Survey (Appendix

Table 3). Across the three groups, the majority of patients were white and female, and about

a quarter were dual eligible with similar comorbidity profiles.”

• It sounds like the accreditation visit is intended to improve the standard of care and is an

intervention in its own right. It would be important therefore to compare hospitals on a like for like time

period either before an accreditation visit or a suitable time after an accreditation visit.

Thank you for this comment.

Dr. Jena and colleagues looked specifically at whether hospitals undergoing the TJC survey had better

outcomes during the week of the survey. They found significantly lower mortality rates for patient

admitted to hospitals during TJC survey weeks compared to those admitted during non-survey weeks

(three weeks before and three weeks after the survey week). Therefore, while there was lower

mortality during the weeks of the survey, the mortality rates returned to baseline after the survey

was complete.

The primary goal of our study is to understand not the short-term effects of inspectors arriving at the

hospital – but whether, at a national level – different approaches to accreditation (private, expensive

accreditation that is quite exhaustive versus a lighter private accreditation versus a state survey) are

associated with differential patient outcomes.

As mentioned above, we performed an additional sensitivity analysis using the updated 2015

Medicare data on a subset of hospitals, specifically those that were deemed in 2014 only (Table 3,

Table 4). Our sample size decreases significantly by including only a subset of hospitals. The results are

qualitatively similar although we do find in this one subset of hospitals that surgical mortality is lower

for those at TJC vs non-TJC. We have included this in our revised manuscript (Appendix Table 8 and 9).

• The committee thought your conclusions may be too strong for the data presented.

Thank you – we have modified our conclusions to reflect the limitations of our data.

• Causal inference should not be made from observational data.

Thank you – we have modified the wording of our discussion.

8

• Confidence intervals for the differences between AO and State Survey and between TJC and non-

TJC in tables 2-5 should be given. The confidence limits should be considered in the interpretation of the

results.

Thank you for this suggestion. Table 2-5 have been updated to include the differences with their

confidence intervals.

First, please revise your paper to respond to all of the comments by the reviewers. Their reports are

available at the end of this letter, below. Please also respond to the additional comments by the

committee.

In your response please provide, point by point, your replies to the comments made by the reviewers and

the editors, explaining how you have dealt with them in the paper.

Thank you for the opportunity to respond to comments and suggestions by the committee and

reviewers to improve our manuscript.

9

** Comments from the external peer reviewers**

Reviewer: 1

Recommendation:

Comments:

Please note that I am at the same university as the study authors and have collaborated with study

authors Jha and Orav on previous studies. However, I have expertise in analysis of Medicare claims data,

have linked TJC accreditation data to Medicare data in a previous study (other than the study I am

currently reviewing and my own study that is referenced, I am unaware of prior studies that perform this

linkage), and (through my prior work) am familiar with the broader literature on the role of hospital

accreditation in influencing patient outcomes.

This article studies the relationship between hospital accreditation and patient outcomes in U.S.

hospitals. The hospital accreditation business is large and is an important mechanism through which

hospitals attempt to ensure the provision of high quality health care. The study links data on Medicare

beneficiaries who are hospitalized for a set of general medical or surgical conditions to data on

accreditation by either the Joint Commission (TJC), other non-TJC accrediting organizations, or state

surveyors. The basic finding of the article is that accreditation bears very little relationship with 3 main

patient outcomes: 30 day mortality, 30 day readmissions, and patient experience with hospital care.

Process measures of quality were not assessed.

I am unaware of other studies which comprehensively analyze the relationship between hospital

accreditation, in particular accreditation by TJC, and patient outcomes, at least using data as recent as

that in this article. At the time of my own work on this issue - which studied how patient outcomes

change during the exact dates of TJC hospital inspections - my review of the literature was consistent

with how these study authors have interpreted current studies. Findings on the role of accreditation in

hospitals are mixed, usually focus on process measures of outcomes, and do not study the relationship

with patient outcomes. I would add that most if not all previous studies are smaller in size than the

current study. The question that's asked is important given the size of this industry, the importance that

is placed by regulators on hospital accreditation, the efforts that hospitals go through to gain

accreditation, and the belief that accreditation is necessary, despite many other factors that may be as

or more important to hospitals in incentivizing them to provide high quality care.

Thank you for your review and the opportunity to respond to your comments to improve our

manuscript.

I have several comments / suggestions:

1. The BMJ audience is international. While the existing Discussion briefly highlights work on the

role of hospital inspection in other countries, many readers of the BMJ will be unfamiliar with how

hospital accreditation works in the US and what is the role and importance of TJC. US reader will know

this information but other ex-US readers may not. I would include more detail on these issues to make

them salient to international readers.

Thank you for this comment. The editorial committee had similar suggestions and we have now added

additional information and references to the introduction.

10

2. I generally found the empirical approach to be appropriate. An alternative approach with

longitudinal data could be to look at how hospital outcomes change before and after accreditation. Aside

from issues with data availability, this analysis may be difficult to conduct if most hospitals stay

accredited and rarely is it the case that hospitals switch in and out of accreditation status.

Thank you for the comment. We agree with the reviewer that a longitudinal analysis with time-

dependent covariates would have been ideal – but as the reviewer points out, few hospitals change

their exposure (type of accreditation) and when they do, it is usually for other reasons that are likely

correlated with outcomes. Therefore, such a strategy would not give us reliable answers in this

instance.

3. The authors can consider adding more discussion as to why accreditation is not associated with

better outcomes. There are at least a few reasons that come to mind. US hospitals compete with one

another within local or regional markets. A primary thesis of competitive markets is that competition

drives improvements in quality. While there are asymmetries of information in health care, one reason

accreditation may not be associated with patient outcomes is that hospitals are already competing on

presumably patient outcomes and other factors. In addition to this explanation, the malpractice system

in the US may already exert influence on hospitals to provide high quality care. Finally, insurers have

information about quality of care in hospitals and may use their bargaining power to influence decisions

of hospitals to invest in quality of care. Insurers can exclude low quality providers from their network and

this competitive pressure may induce investments into quality. Finally, it may be the case that

accreditation can improve process measures, but these measures are poorly correlated with the

outcomes the authors study.

Thank you for these comments. We have now added additional information to our discussion to

address these important points.

“It is possible that accreditation by an independent AO is not associated with better patient

outcomes because the focus of AOs has been on improving structural factors and clinical

processes but less on actually improving patient outcomes. Prior work has shown that efforts

at improving clinical processes of care can lead to better patient outcomes,52,53

but this does

not always hold true.54,55

Furthermore, general hospital accreditation has demonstrated mixed

results on patient outcomes.1,29,31,56

Additionally, we did not observe better patient experience

among patients receiving care at an accredited hospital, and in fact, satisfaction was slightly

worse. This is consistent with findings from Sack and colleagues who showed that

accreditation is not associated with better quality as perceived by patients across 73 hospitals

in Germany.57

Again, this may be a result of accreditation by AOs emphasizing measures that

do not directly translate to better patient experience. There are several other explanations for

why accreditation is not associated with better outcomes. Since U.S. hospitals compete within

local or regional markets, competition may be a driver of overall improvement in quality. It is

also possible that the U.S. malpractice system may be exerting influence on hospitals to

provide high quality care. Finally, it is possible that insurance companies have information

about quality of care in hospitals and may use their bargaining power to influence decisions of

hospitals to invest in quality of care. They might exclude low quality providers from their

network, providing additional pressure for hospitals investments into quality.”

11

4. Can the authors provide some data on how many hospitals attempt to get TJC accreditation but

do not? If this information is available, they might consider comparing those hospitals that achieved TJC

accreditation versus those that attempted to but did not.

This is an interesting question and the other reviewer asked a similar question. It turns out that a very

small proportion of hospitals are denied accreditation – based on the Joint Commission website, it

appears that in 2015, only 8 hospitals were denied accreditation and another 14 received

accreditation with follow-up. It appears that almost all hospitals are accredited.

To respond to the reviewer’s suggestion, we ran new analyses that compared hospitals that received

accreditation by TJC, those that were initially denied but accredited with follow-up and those that

were denied altogether. We see little difference across the groups though to some extent, the denied

group appears to have the lowest mortality rates (though this is likely an artifact of small numbers).

There are no significant differences in mortality and readmissions between these three groups.

Table 5. Risk-Adjusteda Overall and Star Rating Scores for TJC-Hospitals Accredited vs. Accredited with

follow up vs. Denial of Accreditation (2015 Medicare data)

Accredited

[95% CI]

Accredited with

Follow up

[95% CI]

Denial

[95% CI] p-value

b

30-Day Mortality

Medical 10.3%

[10.22-10.36%] 11.2%

[10.11-12.35%] 9.5%

[8.58-10.33%] 0.09

Surgical 2.2%

[2.12-2.21%] 1.8%

[1.03-2.49%] 1.6%

[0.60-2.65%] 0.38

30-Day Readmissions

Medical 22.3%

[22.18-22.41%] 23.6%

[22.44-24.65%] 22.8%

[21.17-24.45%] 0.13

Surgical 15.2%

[14.91-15.43%] 15.3%

[12.54-18.13%] 15.0%

[10.57-19.38%] 0.99


5. Some of the language in the discussion is causal or approaches causal statements. Given that the

study design isn't quasi-experimental, it would be worth while in any revision for the authors to pay close

attention to emphasizing the association of accreditation with patient outcomes, rather than a causal

effect.

Thank you for this suggestion. We completely agree with the reviewer and have now modified our

language to emphasize the association of accreditation with patient outcomes rather than imply a

causal effect.

12

Reviewer: 2

Dear authors,

Thanks for an interesting study on accreditation. The study is really appreciated in the effort to illuminate

the effectiveness of accreditation on patient outcomes relevant to many readers around the world.

However, this review have raised a number of concerns that the authors profitably could address to

strengthen the presented study.

Thank you for your helpful comments and for the opportunity to address the comments.

Introduction:

The introduction could benefit from a detailed summary of the existing literature review on the results of

studies on accreditation and patients’ outcome including a description of methodological challenged in

accreditation research performed so far. Furthermore, the authors’ could profitably expand their focus to

research on other accreditation program with similar content and survey methodology than the TJC and

non-TJC.

The authors’ state that no contemporary accreditation data have shown better outcomes and refer to a

study from 2002. However, other more recent studies do not support this argument:

Azagury D, and Morton JM2. “Bariatric Surgery Outcomes in US Accredited vs Non-Accredited

Centers: A Systematic Review.” J Am Coll Surg. 2016 Sep;223(3):469-77.

Gratwohl A,et al, Use of the quality management system "JACIE" and outcome after

hematopoietic stem cell transplantation. Haematologica. 2014 May;99(5):908-15.

Thank you for your comment. We agree that there are several different types of accreditation with

some focusing more narrowly on certain conditions and others that are broader, focusing on entire

hospitals such as The Joint Commission. As the reviewer points out, there have been more recent

papers on accrediting organizations that do not accredit hospitals but rather specific aspects of the

hospital. While our overarching goal was to focus on hospital-level accreditation, we have updated

our introduction to be more general and include these references about department-specific or

condition-specific accreditation. To be more comprehensive with our literature search, we have

included additional references as well.

“There are many existing types of healthcare accreditation that are condition- or specialty-

specific to hospital- and organization-level efforts. The current literature on accreditation

reveals a mixed story of whether accreditation improves processes of care and outcomes2,3,12–

21,4–11. For hospital accreditation specifically, much of the evidence to date has focused

narrowly on the impact of accreditation on structural factors or processes of care, many of

which are emphasized and assessed by TJC.4,6–11,14,22,23

”

Study question:

There is a discrepancy between the phrasing of the third question for patient experience and the

presented results. Thus, two analyses are performed with first merging the accredited hospitals into one

exposure group (AO’s) compared to state surveys and secondary the TJC and non-TJC are assessed

separately.

Thank you for pointing this out. We have changed the wording of the third question to reflect the

presented results.

13

“And finally, how does patient experience differ between hospitals accredited by an AO

versus those undergoing a State Survey and between hospitals accredited by TJC versus non-

TJC organizations?”

Methods:

A paper on accreditation requires to incorporate a detailed description of the intervention under

investigation in order for the readers to understand the potential mechanisms on how and why

accreditation may affect patient outcome. This includes among others details on content of the

standards and survey methodology for both the TJC and the non-TJC’s frameworks. Furthermore, an

accreditation programme have several aims besides improving patient outcomes, thus a clarification on

what part of the programs that are most likely to affect patient outcomes is needed.

Thank you for this comment. The editorial committee and the other reviewer also commented on the

need for further information on the accreditation process. Please see above for details but we have

now included additional information and references to help the broad swath of BMJ readers, some of

whom may be less familiar with the accreditation process.

We agree with the reviewer that a hospital accreditation program may have aims beyond assessing

and improving patient outcomes, although surely assuring a high level of quality is near the top of

that list. The accreditation programs such as the Joint Commission primarily measure structural and

process measures during their survey visits as a proxy for high quality health care being delivered at

their hospital. We have attempted to clarify that while structural factors and processes are primarily

measured, the data is mixed on whether even those are improved with accreditation, and that for our

study, we wanted to focus on patient outcomes and experience.

It is an important area of concern that the information on accreditation status and mortality and

readmission were not collected within the same time period. Accreditation status is collected between

2014 and 2017 and outcome data between January 1, 2014 and November 30, 2014. The authors’ need

to explain how hospitals that undertake surveys in 2016 or 2017 are likely to perform better on patient

outcomes in 2014?

Thank you for this important comment, which has been echoed above by the editorial committee.

Please refer to our full discussion above. In brief, we thought that our approach was appropriate

because most hospitals are accredited for long periods of time and prior studies suggest that there

aren’t major changes in outcomes after an accreditation visit. In fact, the Joint Commission and

others argue that the date of accreditation is meant simply as a test in what is largely a broader, long-

term quality assurance program. And, since most hospitals retain their accreditation after each visit, a

hospital accredited in 2016 was almost surely also accredited in 2013 and therefore, the 2016 date is

simply a proxy for this being a Joint Commission accredited hospital over the longer run.

This all said, we performed two additional sensitivity analyses to address the reviewer’s concerns.

First, we have now obtained 2015 Medicare data and have repeated our analyses. We find that results

are similar to the original paper submission (Tables 1, 2 above). Second, we then narrowed our

hospitals to those deemed in 2014 to look at 2015 outcomes (Tables 3, 4 above). In that subset of

hospitals, there was no difference in mortality and readmissions for privately accredited hospitals

compared to those undergoing a state survey. There was significantly lower surgical mortality for

hospitals accredited by TJC vs non-TJC entities.

14

To my understanding, the identification of patients and data on mortality and readmission are derived

from Medicare inpatients. Because the study is relevant to readers worldwide, please include a

description of the characteristics of patients eligible for Medicare and possible a reference for further

description. In addition, it is not clear, whether outcomes are restricted to in-hospital data or

independent of place of death and if readmission are to the hospital for the index admission or any

hospital.

Thank you for your comment. We have incorporated the following information into the manuscript

with additional references:

“In the U.S., Medicare is available for people age 65 or older, younger people with disabilities,

and people with end-stage renal disease.30,31

Patients with Medicare often have multiple

chronic conditions and lower median income than the rest of the population.32

”

Patients are selected based on inpatient admission into a hospital, but 30-day mortality is not

conditional on place of death. 30-day readmissions are calculated based on readmission to any

hospital after the index admission. We have added additional text into the manuscript to clarify this:

“Thirty-day mortality is calculated for any death (in-hospital or elsewhere) in the 30 days

following the admission date.”

A description of the collection of data on patient experience should be included in the Methods section.

As clarified in the statistical section, data are collected from April 2015 to March 2016, but it unclear

when data on patients’ experience are gathered. Are “discharge patients” asked on the exact date of

discharge or how is this done in HCAPHS? The paper lacks information on the data collection including

methods for distributing the questionnaire (paper or electronic), whether the procedure were

standardized and therefore similar for all hospitals and finally, by reporting patient response rates

according to exposure groups.

Thank you – we have now included additional information and references about the HCAPHS survey

as well as patient response rates by exposure group.

“The HCAHPS survey is a standardized set of questions given to patients who were discharged

from the hospital who had at least one overnight stay in the hospital as an inpatient, had a

non-psychiatric principal diagnosis at discharge, and were alive at time of discharge. Patients

cannot be surveyed while they are still in the hospital. Sampled patients are surveyed

between 48 hours and 6 weeks after discharge.33

There are four approved modes of

administration, including mail only, telephone only, mail followed by telephone, and active

interactive voice response.34

The HCAHPS survey asks discharged patients 27 questions about

their recent hospital stay, and these questions are grouped and reported in the following 11

publicly reported measures: four composite measures of clinical domain (responsiveness of

hospital staff, pain management, discharge information, and care transition), three

communication measures (with physicians, with nurses, and about medications), two items

related to hospital environment (cleanliness and quietness), and two global measures (overall

hospital rating and likelihood to recommend). The CMS summary star rating scores hospitals

on a one to five-star scale based on the eleven domains in the HCAHPS survey. Response rates

for the three groups were 29% for TJC, 30% for non-TJC, and 34% for State Survey.”

15

The specific diagnostic codes included for each medical condition needs to be outlined as the groupings

are not exhaustive e.g. “Metabolic diseases” or “respiratory diseases” the later do obviously not include

COPD. Furthermore, their relevance for 30-day mortality must be presented in more details too.

Thank you for the suggestion. We have updated our Appendix Table 1 in our manuscript to include the

specific diagnostic codes for each condition. We used 30-day mortality as it is a metric collected by

the Centers for Medicare and Medicaid (CMS) and used in numerous studies both in the U.S. and

internationally to evaluate patient outcomes.

Appendix Table 1. Common medical and surgical procedures and their associated diagnosis-related

group (DRG) codes

Condition Code Code Description

Medical Condition

Acute Myocardial

Infarction

280 ACUTE MYOCARDIAL INFARCTION, DISCHARGED ALIVE W

MCC

281 ACUTE MYOCARDIAL INFARCTION, DISCHARGED ALIVE W

CC

247 PERC CARDIOVASC PROC W DRUG-ELUTING STENT W/O

MCC

282 ACUTE MYOCARDIAL INFARCTION, DISCHARGED ALIVE

W/O CC/MCC

246 PERC CARDIOVASC PROC W DRUG-ELUTING STENT W MCC

OR 4+ VESSELS/STENTS

249 PERC CARDIOVASC PROC W NON-DRUG-ELUTING STENT

W/O MCC

283 ACUTE MYOCARDIAL INFARCTION, EXPIRED W MCC

Chronic Heart

Failure

292 HEART FAILURE & SHOCK W CC

291 HEART FAILURE & SHOCK W MCC

293 HEART FAILURE & SHOCK W/O CC/MCC

Pneumonia

194 SIMPLE PNEUMONIA & PLEURISY W CC

193 SIMPLE PNEUMONIA & PLEURISY W MCC

195 SIMPLE PNEUMONIA & PLEURISY W/O CC/MCC

Chronic Obstructive

Pulmonary Disease

190 CHRONIC OBSTRUCTIVE PULMONARY DISEASE W MCC

191 CHRONIC OBSTRUCTIVE PULMONARY DISEASE W CC

192 CHRONIC OBSTRUCTIVE PULMONARY DISEASE W/O

CC/MCC

Stroke

65 INTRACRANIAL HEMORRHAGE OR CEREBRAL INFARCTION

W CC


W MCC


W/O CC/MCC

Sepsis

871 SEPTICEMIA OR SEVERE SEPSIS W/O MV 96+ HOURS W MCC

872 SEPTICEMIA OR SEVERE SEPSIS W/O MV 96+ HOURS W/O

MCC

Esophageal/Gastric

Disease 392

ESOPHAGITIS, GASTROENT & MISC DIGEST DISORDERS W/O

MCC

16

391 ESOPHAGITIS, GASTROENT & MISC DIGEST DISORDERS W

MCC

372 MAJOR GASTROINTESTINAL DISORDERS & PERITONEAL

INFECTIONS W CC

371 MAJOR GASTROINTESTINAL DISORDERS & PERITONEAL

INFECTIONS W MCC

Gastrointestinal

Bleeding

378 G.I. HEMORRHAGE W CC

377 G.I. HEMORRHAGE W MCC

379 G.I. HEMORRHAGE W/O CC/MCC

Urinary Tract

Infection

690 KIDNEY & URINARY TRACT INFECTIONS W/O MCC

689 KIDNEY & URINARY TRACT INFECTIONS W MCC

Metabolic Disease 641 NUTRITIONAL & MISC METABOLIC DISORDERS W/O MCC

640 NUTRITIONAL & MISC METABOLIC DISORDERS W MCC

Arrhythmia

310 CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W/O

CC/MCC

309 CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W CC

308 CARDIAC ARRHYTHMIA & CONDUCTION DISORDERS W

MCC

243 PERMANENT CARDIAC PACEMAKER IMPLANT W CC

244 PERMANENT CARDIAC PACEMAKER IMPLANT W/O CC/MCC

Chest Pain

313 CHEST PAIN

0

287 CIRCULATORY DISORDERS EXCEPT

Renal Failure 683 RENAL FAILURE W CC

682 RENAL FAILURE W MCC

Respiratory Disease 177 RESPIRATORY INFECTIONS & INFLAMMATIONS W MCC

178 RESPIRATORY INFECTIONS & INFLAMMATIONS W CC

Hip Fracture

481 HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W CC

470 MAJOR JOINT REPLACEMENT OR REATTACHMENT OF

LOWER EXTREMITY W/O MCC

482 HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W/O

CC/MCC

480 HIP & FEMUR PROCEDURES EXCEPT MAJOR JOINT W MCC

Surgical Procedure

Coronary Artery

Bypass Grafting

236 CORONARY BYPASS W/O CARDIAC CATH W/O MCC

234 CORONARY BYPASS W CARDIAC CATH W/O MCC

233 CORONARY BYPASS W CARDIAC CATH W MCC

235 CORONARY BYPASS W/O CARDIAC CATH W MCC

Open Abdominal

Aortic Aneurysm

Repair

238 MAJOR CARDIOVASC PROCEDURES W/O MCC

237 MAJOR CARDIOVASC PROCEDURES W MCC OR THORACIC

AORTIC ANEURYSM REPAIR

Endovascular

Abdominal Aortic

Aneurysm Repair

238 MAJOR CARDIOVASC PROCEDURES W/O MCC

Colectomy

330 MAJOR SMALL & LARGE BOWEL PROCEDURES W CC

329 MAJOR SMALL & LARGE BOWEL PROCEDURES W MCC

331 MAJOR SMALL & LARGE BOWEL PROCEDURES W/O CC/MCC

Pulmonary

Lobectomy

164 MAJOR CHEST PROCEDURES W CC

165 MAJOR CHEST PROCEDURES W/O CC/MCC

17

163 MAJOR CHEST PROCEDURES W MCC

Hip Replacement 470 MAJOR JOINT REPLACEMENT OR REATTACHMENT OF

LOWER EXTREMITY W/O MCC

Finally, the Methods section would be strengthened by a section that clarifies why it is appropriate to

compare hospitals that are accredited with those that are not. In particularly when Table 1 outlines

major differences in size (resulting in huge difference in numbers of hospitalization pr. hospitals; state

survey average 171 vs TCJ = 1253 and Non-TJC=1006), teaching status, urban, Critical assess hospital,

and numbers of intensive care units. The later may have a significant influence on the patient’s chance of

survive.

We agree that there are important baseline differences in hospitals that chose to undergo

accreditation by a private organization such as the Joint Commission (TJC) and others (non-TJC),

versus those hospitals that opt to undergo the complimentary state survey.

Our initial hypothesis was that since hospitals that undergo private accreditation through TJC or non-

TJC organizations are often bigger hospitals with more resources, our results would be biased toward

showing better outcomes at privately accredited hospitals (and at TJC hospitals over non-TJC privately

accredited hospitals). Given that we did not see this difference, we have more confidence in the

finding that there seems to be little advantage of getting a private accreditation or in TJC over other

approaches.

Statistical analysis:

The authors need to clarify the terms dual-eligibility, and CAH status – what is a Critical Assess Hospital

for those not familiar to the US hospital system). Furthermore, it is unclear what population is included

for the outcome of readmission including how competing risk for death is handled in the analyses?

Thank you for this suggestion. We agree the need to clarify several of our terms and have now

incorporated the following into our manuscript:

”Medicaid eligibility was determined using the State Buy-In Coverage Count variable. Any

beneficiary with at least 1 month of state buy-in was considered Medicaid eligible. Dual-

eligibility refers to patients who are eligible for both Medicaid and Medicare. Eligibility for

Medicaid is determined primarily by state-level poverty thresholds. Information on hospital

characteristics was obtained from the American Hospital Association (AHA) annual survey and

Medicare Impact File. Admissions to non–acute care hospitals, federal hospitals, and those

outside the 50 states and the District of Columbia were excluded. Critical Access Hospitals

(CAHs) were included in our study and are defined by statute in the United States. Their key

characteristics are that they are small (have 25 or fewer inpatient beds) and are rural (located

more than 35 miles from another hospital). These rural hospitals face substantial burdens in

providing access to high quality care. To help reduce their financial burden, these hospitals are

reimbursed on a cost basis.45

”

“We followed the standard approach to calculating readmissions as developed by CMS.50

The

numerator includes any unplanned readmissions to a non-federal, short-stay, acute-care or

18

critical access hospital within 30 days after discharge. Multiple readmissions are counted

once, and same-day readmissions for the same principal diagnosis at the same hospital are

excluded. The denominator includes beneficiaries 65 years or older who are hospitalized at

non-federal, short-stay, acute care or critical access hospitals. A readmission may also serve as

an index hospitalization. There are numerous exclusion criteria which include: death during

admission, discharge against medical advice, cancer hospitalization, or lacking continuous

enrollment in Medicare for at least 30 days post discharge.”

For calculating readmissions, we followed CMS guidelines, which do currently do not address

competing risk. Our mortality measure uses medical and surgical conditions as a selection mechanism

and to adjust for differences in patient severity but does not assign mortality to any one cause.

Similarly, our readmission measure excludes from the denominator those who died during the

admission.

The study would be strengthened considerably by adding a number of stratified analyses to explore the

results for different exposure groups. This includes e.g. stratification by calendar periods, level of

compliance achieved by the hospitals (accredited, accredited with excellence etc.) and how many cycles

of accreditation the hospitals have completed. Furthermore, it could be interesting to see the result for a

sensitivity analysis restricting to hospitals with ICU.

Thank you for this suggestion. We have performed several additional sensitivity analyses as suggested

by the committee and reviewer. Above, we have performed the following analyses:

- By calendar period (2015 outcomes data for hospitals deemed in 2014; Tables 3, 4 above;

manuscript Appendix Tables 6, 7)

- By level of compliance achieved by the hospitals (only available for TJC-accredited hospitals;

Table 5 above; manuscript Appendix Tables 8, 9)

- By hospitals with an ICU (Tables 6, 7)

Unfortunately, there is no publicly available data regarding how many cycles of accreditation hospitals

have completed.



with an ICU Accredited by an Accrediting Organization (AO) vs. Reviewed by a State Survey Agency

(2015 Medicare data)

AO

% [95% CI]

State Survey

% [95% CI]

Difference

% [95% CI] p-value

b

30-Day Mortality

Medical 8.6%

[8.55-8.70%]

8.9%

[8.27-9.43%]

-0.23%

[-0.82-0.37%] 0.46

Surgical 1.7%

[1.62-1.72%]

1.6%

[1.19-2.00%]

0.07%

[-0.34-0.48%] 0.72

30-Day Readmissions

Medical 22.5%

[22.40-22.67%]

22.4%

[22.50-24.34%]

-0.89%

[-1.82-0.05%] 0.06

Surgical 13.3%

[13.04-13.54%]

14.3%

[12.16-16.44%]

-1.01%

[-3.16-1.13%] 0.35

19




with an ICU Accredited by TJC vs. non-TJC (2015 Medicare data)

TJC

% [95% CI] Non-TJC


% [95% CI] p-value

b

30-Day Mortality

Medical 8.6%

[8.54-8.70%] 8.4%

[8.17-8.72]

0.18%

[-0.11-0.47%] 0.22

Surgical 1.7%

[1.61-1.72%] 1.7%

[1.51-1.90%]

-0.04%

[-0.25-0.18%] 0.72

30-Day Readmissions

Medical 22.5%

[22.37-22.66%] 22.6%

[22.14-23.01%]

-0.07%

[-0.54-0.40%] 0.78

Surgical 13.3%

[13.04-13.57%] 13.1%

[12.24-14.02%]

0.17%

[-0.79-1.13%] 0.72


Results:

The results section would benefit from more elaboration on the absolute differences in outcomes

between exposure groups and the stratified analyses mentioned above.

Thank you for this suggestion. We have now added columns to our tables of the differences in the

outcomes.

Discussion:

Regarding line 14 page 15 “given that…” to my understanding, the results are adjusted for age, sex and

so forth. Please clarify, how patients’ age and economic status adds into this argument? Furthermore,

there is no evidence in the study to that patients in the state survey hospitals are poorer – if this is the

case, it must be included in the multivariable models too. Furthermore, the claim regarding cost-

effectiveness of accreditation is not substantiated by data presented in the manuscript. The authors

should avoid making conclusions beyond presented data.

The reviewer is correct that we have adjusted our results for patient and hospital characteristics. We

agree with the reviewer about cost-effectiveness and have now deleted these sentences from our

manuscript.

Line 31 page 15 “prior work..” the authors must provide a more nuanced perspective on this matter, as

other studies have shown that providing patients with the relevant processes of care improves their

outcomes, including 30-day mortality – here are some of the work from Denmark in favor of processes on

outcomes.

Kristensen PK et al, ”Are process performance measures associated with clinical outcomes

among patients with hip fractures? A population-based cohort study.” Int J Qual Health Care. 2016 Dec

1;28(6):698-708

20

Ingeman A et al, “Quality of care and mortality among patients with stroke: a nationwide follow-

up study.” Med Care. 2008 Jan;46(1):63-9.

We agree and have now made changes to our manuscript to point out that the data of whether

improved process measures lead to better outcomes has shown mixed results. This is now both in the

introduction (as mentioned above) and the discussion. We had added these additional references to

our paper.

“Prior work has shown that efforts at improving clinical processes of care do not always can

lead to better patient outcomes,51,52

but this does not always hold true.53,54

Furthermore,

general hospital accreditation has demonstrated mixed results on patient outcomes.1,30,32,55

”

Line 56 page 15 “we assumed..” Please provide data or references to support the argument that TJC-

accredited hospitals are the most financially well-resourced and in addition, that finances alone can

contribute to improved patient outcomes.

Thank you for this comment. We agree that our discussion could use more clarification and a stronger

basis with references. As discussed in the introduction, TJC accreditation is expensive, not only due to

the direct costs but also the indirect costs. We are hypothesizing that hospitals with the resources and

capital to spend on hospital accreditation would not only sought out accreditation from the “gold

standard” TJC, but also had the resources and capital to direct toward improving patient safety and

quality of care. We have updated our discussion to clarify that this is our hypothesis on TJC-accredited

hospitals.

“We hypothesized that the most financially well-resourced hospitals in the country may have

sought out TJC accreditation – and since those same hospitals may have resources and capital

to dedicate toward improving patient care and outcomes, they might have better outcomes

(whether due to TJC accreditation or not). In our study, we did not find an association

between accreditation status and patient outcomes.”

Line 38 page 16 “our work..” this section need to include a discussion of how the presented result can be

compared to the referenced studies (reference 15 and 30). A correction, reference 30 includes data from

31 hospitals. If reference 30 is discussed it could be relevant to discuss another study on acute

readmissions from the same study group. In addition, the relevance of discussing papers on process

performances measures need to be explained (reference 31 and 22).

Thank you for this comment. We agree that this section could use enhancement of how our results

compared to the prior literature. To this end, we have updated our discussion. Furthermore, we have

also corrected reference 30 and added the study on readmissions from the same group (Falstie-Jensen

et al).

“Our work adds to a limited and mixed body of evidence on accreditation and outcomes.

Griffith et al. found that lower quality scores in US acute care facilities were associated with

higher mortality rates,15

while Falstie-Jensen et al found that in 31 Danish hospitals, full

accreditation – versus partial accreditation – was associated with lower mortality rates and

shorter length of stay but no difference in acute readmissions.57,58

Schmaltz and colleagues

have shown that accredited hospitals have marginally better scores on process measures,35

but

Bogh et al. found that accreditation was not associated with larger improvement for patients

21

with acute stroke, heart failure, or ulcers.41

Furthermore, there has been wide variation in

mortality reported between top and bottom decile accredited hospitals.59

Barnett and

colleagues found that unannounced accreditation visits by TJC lead to improved mortality for

patients admitted during that week but that the mortality rates then returned back to

baseline.60

These studies provide important information on how accreditation is associated

with process and outcomes across the globe. There is less information about how accreditation

in general, and specifically by the TJC, impacts patient outcomes across common medical and

surgical conditions as well as patient experience over a longer period of time.”

Line 17 page 17 second paragraph: the section would be stronger using fewer assumptions and more

facts including a clear discussion of the risk of selection and information bias and confounding related to

the study.

Thank you for the comment. We agree that this section could use exactly the modifications suggested

and have included a clear discussion of the risk of selection and information bias and confounding

related to the study.

“Our study has several limitations. First, as this is an observational study, we cannot assess

causality. Due to the non-randomized study design, we cannot exclude the possibility that our

results may be influenced by confounding by unmeasured factors. There is also the possibility

of selection bias of which hospitals decide to undergo accreditation. Given that accreditation

is a choice, one would assume that the institutions with better and more resources would

undergo this process, potentially biasing us towards finding a benefit of accreditation. Second,

our mortality and readmissions measures are calculated only on the Medicare population, and

we do not know if our findings are similar among commercially-insured or other publicly-

insured populations. The rates are also based on administrative claims data, which lacks

detailed clinical information. However, CMS depends on this data to assign rankings and

distribute payment determinations in national pay-for-performance programs. Moreover,

when examining patient experience, HCAHPS survey has a low response rate, and responses

are subjective and affected by personal and cultural expectations. Finally, our exposure in this

study is whether or not hospitals were accredited, and if so, by which accrediting body. Risk of

information bias is minimized due the use of national databases. We did not take into account

when the accreditation or review occurred in the last three years, as this is a cross-sectional

study.”

Conclusion:

Accreditation by TJC is perhaps the main choice but this does not automatic make it a gold standard.

Please explain how the study results substantiates the last sentence.

We completely agree with the reviewer that TJC accreditation should not automatically be considered

the gold standard without evidence. We have now reworded our sentence to the following:

Further, we found that accreditation by TJC, which is the most common form of hospital

accreditation, appears to confer no additional benefit for patients over other lesser known

accrediting agencies.

Ashish K. Jha, MD, MPH · Ashish Jha, MD, MPH Ashish K. Jha, MD, MPH Director, Harvard Global...

Documents

Transcript of Ashish K. Jha, MD, MPH · Ashish Jha, MD, MPH Ashish K. Jha, MD, MPH Director, Harvard Global...