Holding a mirror up to two industries

We were awarded a Mirror Award – hosted by Syracuse U. Newhouse School of Communications for those “who hold a mirror to their own industry for the public’s benefit.”

Lessons to improve our reporting on evidence: An analysis of 1,800+ stories over 7 years

Gary SchwitzerPublisher

MIT Medical Evidence Boot CampDecember 4, 2012

Infoxication

The editors of the journal PLoS Medicine wrote: “Schwitzer’s alarming report card of the trouble with medical news stories is thus a wake-up call for all of us involved in disseminating health research-researchers, academic institutions, journal editors, reporters, and media organizations-to work collaboratively to improve the standards of health reporting.” Susan Perry, on her MinnPost.com column referred to HealthNewsReview.org as: “indispensable to consumers & journalists”

Dan Gillmor, director of the Knight Center for Digital Media Entrepreneurship at Arizona State University’s Walter Cronkite School of Journalism and Mass Communication: “Calm and thorough analysis of health news journalism from HealthNewsReview.org”

Dr. Steven Kussin, Director of The Shared Decision Center, wrote: “All doctors should suggest patients register at HealthNewsReview.org.”

Science journalist Paul Raeburn on the Knight Science Journalism Tracker:“I’m beginning to think that Schwitzer’s criteria for judging stories ought to be printed on wallet cards for reporters, like Miranda warnings, to remind them what questions to ask. I could use one of those myself.”

What others are saying about us

Our criteria: Does the story explain…

• What’s the total cost?• How often do benefits occur?• How often do harms occur?• How strong is the evidence?• Is the condition exaggerated?• Is this really a new approach?• Is it available?• Are there alternative choices?• Who’s promoting this?• Do they have a financial conflict of interest?

After 1,800 stories over 6+ years

~70% of stories fail to:

✔ Discuss costs

✔Quantify potential benefits

✔ Quantify potential harms

✔ Evaluate the quality of the evidence

Exaggerate or emphasize benefits

Ignoreor

Minimize Potential Harms

In an election year, when we should have spent more time helping citizens weigh why we spend 17% of GDP on

health care, yet leave 17% of our neighbors uninsured….

…instead, news often painted a kid-in-candy-store picture

of U.S. health care interventions

TerrificRisk-freeWithout a price tag

What CNN didn’t report…

• Resignations, questions about conflicts of interest and fraud probes of the program in question.

• Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.

Robert Bazell: “For the first time ever an experimental drug is showing great promise of slowing the progression of Alzheimer's disease.” NBC News - October 8, 2012

FIRST TIME EVER?

July 17, 2012: New Alzheimer’s Drug Slowing Progression of the Disease – CBS

March 8, 2012: Alzheimer’s treatment in late stages of disease does slow progression (The Independent)

October 20, 2011: Antiviral Drugs May Slow Alzheimer's Progression (Science Daily)

May 11, 2009: Can New Drug Slow Progress of Alzheimer's? (ThirdAge.com)

August 3, 2008: Drugs May Slow Progress Of Alzheimer’s (NPR on two drugs)

5-minute Google search = 6 different approaches reported to slow progression of AD just in past 4 years

Questions you might ask:

“a revised look at two trials that were declared failures” – What does that mean?

How big was potential benefit? AP reported: “Taken separately, the studies on the drug - Eli Lilly & Co.'s solanezumab - missed their main goals of significantly slowing the mind-robbing disease or improving activities of daily living. But pooled results found 34 percent less mental decline in mild Alzheimer's patients compared to those on a fake treatment for 18 months.”

How big were potential harms? Angina…brain swelling…brain bleeds

Common flaws: too much stenography – not independent vetting of studies in journals

Deifying big names/big journals - Publication in a medical journal does not guarantee the findings are true (or even important).

Not ready for prime time – journals meant for conversation among scientists

Never intended to be sources of daily news. So if you’re going to use them that way, you simply must be aware of the landscape:

• retractions, research fraud, fabrication, falsification of data

• unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”)

• ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)

Failure to evaluate inherently weak science

Idolatry of the surrogate – Not understanding or simply not reporting that surrogate outcomes (like tumor shrinkage) may not translate into clinically meaningful outcomes (longer life).

Using causal language to describe observational studies

Exaggerating effect size - Using relative, not absolute risk reduction numbers.

Reckless extrapolation - Predicting what may happen in humans – and soon - based on very preliminary animal / lab science.

Lack of awareness of conflicts of interest & other ethical issues

• commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies

Common flaws (Examples provided later upon request)

NBC: “The science that just might justify an American addiction…Bona fide science. A real study of a lot of people. …There are benefits… may decrease heart disease by as much as 37 percent.”

The piece ended by quoting Mark Twain:

“The only way to keep your health is to eat what you don’t want, drink what you don’t like, and do what you’d rather not.”

We cited another Mark Twain quote – just substitute “network TV news” for “newspaper” – imagine if Twain had lived to see today’s TV news:

“If you don’t read the newspaper, you’re uninformed. If you read the newspaper, you’re mis-informed.”

Misreporting of Observational Studies: Can Scientific Journals Help? (more on journals later)

Two birds killed with one stone in HealthDay story

1. Over-reliance on news releases2. Idolatry of the surrogate

Press release: ”In recent years, drugs that inhibit BRAF activity have rapidly halted and reversed tumor growth in about 90 percent of treated patients, but most patients’ response is temporary, with tumor growth resuming in six or seven months. Investigations into how this resistance emerges have suggested that the MAPK pathway gets turned back on through activation of MEK, another protein further down the pathway.”

Story: “Drugs that inhibit BRAF activity can rapidly stop and reverse tumor growth in about 90 percent of patients. But the response is temporary in most cases, and tumor growth resumes in six or seven months, the researchers explained. Previous research suggested that this drug resistance develops because the MAPK pathway gets turned back on through activation of MEK, another protein that is part off the MAPK pathway.”

---------------------------------------------Quality of the evidence?

The story dutifully reports on the primary outcome of the study, progression-free survival, but doesn’t explain that this may not translate to an overall increase in survival time for patients taking the drug combination.

Our review:

The story fails to point out the intermediate endpoints that look at risk factors for heart attack and stroke (such as cholesterol, and signs of thickening or calcium deposits in arteries) are not what we care about, which is whether heart attacks and strokes were any more or less common. Since it takes longer to detect such events, the researchers chose to look at markers for future disease rather than cardiovascular events. That’s a legitimate way to explore a hypothesis, but must be emphasized as preliminary. Don’t forget that we got into the widespread but misguided practice of prescribing estrogen for cardiovascular prevention back in the 80s based on just such assumptions, based on data that showed HT had favorable effects on cholesterol and other markers–yet in the end it caused more heart attacks and strokes.

“The numbers don’t lie….or explain anything about what they mean.”

Screening Madness

• Crusading• one-sided• advocacy

I am not anti-screening

• I do oppose imbalanced, incomplete messages on screening tests

• I do oppose messages that emphasize benefits and minimize or ignore harms

• I do oppose messages that only tell anecdotes of those who claim their life was saved by screening – something that can never be proven.

Crossing line from journalism to advocacy

A physician wrote to me after seeing this:

"Could a political reporter say 'Vote for Obama?'!"

“Well, we’ve had a conclusion for many, many years at Susan G. Komen, almost a generation. Screening saves lives. The 5-year survival rates for breast cancer diagnosed early is 98 percent…and this is largely due to screening and early diagnosis.”

Crusading advocacy on CBS News

Left CBS, left “journalism,” just joined ABC

Can you imagine a political reporter wearing a button saying, “Vote for Romney” ?

Crossing line from journalism to advocacy

"For you women out there and for the men who love you, screening matters. Do it. This disease can be completely curable if you find it at the right time."

Katherine O’Brien has metastatic breast cancer – on her ihatebreastcancer blog:

“Early detection is not a cure. ‘Completely curable’ is a like a fat man wearing a hockey jersey. It covers a lot of ground.”

This is what he wrote:

The question was simple enough.

"Do you want to be scanned?" asked the hospital PR woman.

Um, sure?

(More on Minneapolis Star Tribune later)

“All screening programmes do harm; some do good as well.”

- Dr. Muir Gray

Some screening stories we should hear more about….

Track this yourself: • When you disagree with evidence, use politics to attack the process• There has been public ridicule of fact that a woman pediatrician heads USPSTF.• Since when does it require subspecialty in urology or radiology to evaluate evidence?

Leaders of ZERO: The Project to End Prostate Cancer

When the US Preventive Services Task Force recommended against routine screening of men with the PSA blood test, CEO Skip Lockwood said the UPSTF’s advice “condemns tens of thousands of men to die this year and every year going forward…”

COO Jamie Bearse wrote about the chief medical officer of the American Cancer Society:

“Otis Brawley has killed more men by giving them an excuse to not be tested.”

Track this yourselves: Why is it that evidence-based, shared decision-making advocates call for choice but screen-everybody advocates promote mandate not choice?

“Many of these free screenings are designed more to get patients for hospitals and clinics and doctors than they are to benefit the patients. That's a huge ethical issue that needs to be addressed.

We're not against prostate cancer screening. We're against a man being duped and deceived into getting prostate cancer screening."

Brawley on offensive:

Duped & Deceived?

Tim Glynn, lawyer, age 47 when doctor “decided I’d have a PSA test without consulting me.”

“Men should be aware of the truly terrible consequences. As a screening tool, you could do as well by throwing dice on a table.”

Profiled in Shannon Brownlee’s NYT Sunday mag piece, “Can Cancer Ever Be Ignored?”

DECISIONS Study: PSA Decision

• 70% of men reported a discussion before PSA test • MD generally initiated discussion (65%)• MD generally recommended screening (73%) • “Pros” discussed some/a lot in 71% of

discussions, “cons” not discussed at all in 68%• 55% reported being asked their opinion• 48% couldn’t answer any of three knowledge

questions

Hoffman, et al. Arch Intern Med 2009; 169:1611

Prizes for Prostates

The marketing of screening

In 2010 after the National Lung Screening Trial results were released, the ACS posted on its blog:

"It's only been a few days since researchers released preliminary results...(which) although enormously promising…not enough to call for routine use of this screening test, even in heavy smokers. ...But our greatest fear was that forces with an economic interest in the test would sidestep the scientific process and use the release of the data to start promoting CT scans. Frankly, even we are surprised how quickly that has happened."

TheNNT.com: number needed to treat, to screen, to harm

• Began with virtual colonoscopy –after routine physical.

• But more than just colon is captured & CT showed something on kidney, liver, and lungs.

• Kidney and liver issues benign after liver biopsy, PET scan and more CT scans. But lung questions led to major lung surgery, sawing him open through ribs

“I awoke in the recovery room after 5 hours, with a chest tube, a Foley catheter, a subclavian central venous catheter, a nasal oxygen catheter, an epidural catheter, an arterial catheter, subcutaneously administered heparin, a constant infusion of prophylactic antibiotics, and patient-controlled analgesia with intravenously administered narcotics…..Excruciating pain.” • No malignant disease – all “incidentalomas” • Total cost > $50,000

• All precipitated by a screening test

Sharon Begley in Newsweek:

Dr. Stephen Smith, Professor emeritus of family medicine at Brown University School of Medicine, tells his physician not to order a PSA blood test for prostate cancer or an annual electrocardiogram to screen for heart irregularities, since neither test has been shown to save lives. Rather, both tests frequently find innocuous quirks that can lead to a dangerous odyssey of tests and procedures.

Dr. Rita Redberg, professor of medicine at the University of California, San Francisco, and editor of the prestigious Archives of Internal Medicine, has no intention of having a screening mammogram even though her 50th birthday has come and gone. That’s the age at which women are advised to get one. But, says Redberg, they detect too many false positives (suspicious spots that turn out, upon biopsy, to be nothing) and tumors that might regress on their own, and there is little if any evidence that they save lives.

• Reported on woman with DCIS – ductal carcinoma in situ• Reported on her choice to pursue active surveillance rather

than immediate aggressive intervention such as bilateral prophylactic mastectomy

• Reported on the shared decision-making program at UCSF that helped her understand the tradeoffs.

THAT IS GOOD JOURNALISM

Home-cooking on health biz stories

• Minneapolis Star Tribune lags behind smaller paper, smaller market on MSP medical device co. troublesome issues

• Apparent cheerleading on local Medtronic, St. Jude Medical, Boston Scientific stories.

• Search “Star Tribune” on our site to see 4 story examples in a month

• They’re not alone

Journalists could help people understand and deal with the clash between:

• Science• Evidence• Data• Recommendations for

entire population• What we can prove• Grasping uncertainty and

helping people apply critical thinking to decision-making issues

Intuition Emotion Anecdote Decision-making by an

individual What we believe, wish, or

hope Promoting false certainty

where it does not exist

• Surrogate markers may not tell the whole story• Does The Language Fit The Evidence? – Association Versus Causation• Resources for Reporting on Costs of Medical Interventions• “Off-label” Drug Use and Marketing• 7 Words (and more) You Shouldn’t Use in Medical News• Problems with Reporting on News from Scientific Meetings• Absolute vs. Relative Risk• Number Needed to Treat• Commercialism• Single Source Stories• FDA Approval Not Guaranteed• Phases of Drug Trials• Medical Devices• Animal & Lab Studies

Read John Ioannidis to learn pitfalls of a steady diet of journal stories

PLoS Med 2005; 2(8): e124

Journals complicit in miscommunication

• Editors of the HEART Group journals last week stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.”

– J AM COLL CARDIOL, Vol. 60, No. 23, 2012

• “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies”

– Am J Obstet Gynecol, published online 11/29/12

• “Spin and Boasting in Research Articles.”- Commentary in Arch Pediatr Adolesc Med:

[published online October 2012]

JOSH BILLINGS (PEN NAME OF HUMORIST HENRY WHEELER SHAW, 1818 – 1885)

“I honestly believe it is better to know nothing than to know what ain’t so.”

Ask yourselves

• If your beat includes covering studies/clinical news, what is mix of stories just about “new stuff”?

• Do you think you might be reporting too much of this?

• If so, have you spoken with editors about this? Do you want our help?

• If you don’t think you report too much of this, how would you stack up on our 10 criteria?

• If you don’t accept or agree with our 10 criteria, what criteria do you use?

GGary@HealthNewsReview.org

Thank you