…when the conditions of an experiment are once established and worked up according to the mind's...
-
Upload
brady-mckinnon -
Category
Documents
-
view
213 -
download
0
Transcript of …when the conditions of an experiment are once established and worked up according to the mind's...
“…when the conditions of an experiment are once established and worked up according to the mind's preconceived idea, an induced or premeditated observation will, as we said, result.”
“If men discuss and experiment … to prove a preconceived idea …, they no longer have freedom of mind, and they no longer search for truth.”
An experiment done to prove something is different from an experiment done to test something.
A Favorite Example (10/2003):
Patients with uncomplicated DVT/PE
Randomize between
Unfractionated heparin (“standard”)
New drug (easier to use)
Compare rates of DVT, PE, sudden death
Open label, >2,000 patients/subjects
Non-inferiority design
The results:
Similar outcomes in DVT/PE (slight NS advantage to new drug)
Similar outcomes re: mortality
Patients getting new drug tended to leave the hospital earlier (sub-q rather than i.v.)
Conclusion: Non-inferior in efficacy; dosing convenience make it an attractive drug
What’s Wrong?
New drug was compared against old approach of known efficacy
Everybody got treated (no placebo gremlins)
Trial design was standard; sample size was appropriate; few drop-outs
Monitoring was appropriate (Incl. DSMB)
Reputable researchers, sites
The trial was ethical and well-done ... but …
The “standard” therapy was no longer usual practice … drugs similar to the test drug had already largely replaced unfractionated heparin, at least in academic centers
The trial outcome can be restated as “This drug is probably no worse than what we used to do ten years ago”
A strong finding in favor of licensing the drug, but very little help in deciding how to use it wisely
A Non-so-benign Example:
Dan Markingson and his mother, Mary WeissDan’s death in an industry-sponsored clinical trial raises questions about how we do them and monitor them,
Dan’s medical history:
Bright kid; got a B.A. in English from the University of Michigan … sought employment in California.
Over the following three years, developed paranoia, changed his name, adopted cult beliefs.
Indicated to his mother that there was an apocalyptic event approaching, that he might be instructed to kill and would do so.
The story gets complex:
Several maternal interventions fell short
Hospitalized on a hold, given antipsychotic meds; commitment proceedings begun
All examiners agreed that he lacked full decisional capacity and was a danger to himself and others
Result: Stay of commitment
Stay of commitment:
The court finds in favor of commitment, but stays the order on conditions
Typically: “As long as you comply with treatment and do OK with it, you won’t actually be committed and will still have control over your own life”
Conceptually like a revokable parole
No guardian is named
The treatment chosen for Dan:
CAFÉ: Clinical Trial Comparing Atypicals in First Episode
Randomly assign among the three most common “atypical” antipsychotic agents for patients having their first psychotic episodeSeroquel (quetiapine), Zyprexa (olanzapine), and Risperdal (risperidone.)
Study designed, funded by AstraZeneca
Digression: Atypical antipsychotics
All three were licensed and approved;
Very successful drugs commercially, having largely supplanted earlier antipsychotic drugs on the strength of claims that they had fewer autonomic side effects and a lower incidence of tardive dyskinesia;
One of these would very probably have been chosen if not for study entry
How would the study alter Dan’s Rx?
Double-blind;
Limits on other agents; discontinuing or crossing over would require study withdrawal;
Treatment in other respects would be pretty much the same as getting one of the drugs non-randomly out of study.
Concerns/Oddnesses:
Should he be his own decisional authority on study matters?
The trial forbade entry of patients who were suicide risks, but not of patients who were homicide risks(!) Dan had threatened to kill others, but not himself
45 CFR§46.111 (b); 21 CFR§56.111 (b)
The story continues:
Dan’s mother objected to his study enrollment
She was deemed not to have status in the matter (because Dan’s commitment had been stayed, she was not named guardian)
Dan was randomized to receive Seroquel™, and was discharged to a halfway house
The story continues:
Dan seemed improved at his psychiatrist visits, but they were infrequent
Social workers and Dan’s mother recorded concerns that he was still delusional and still at risk to harm himself or others
Dan’s mother asked that he be removed from the study and tried on a different drug - again deemed to have no status
The story continues:
At the end of the six-month stay, his psychiatrist asked that the stay be extended (rather than effecting or vacating the commitment) -- might be at risk if he stops treatment
Mother warned in writing that she feared he would kill himself or someone else; study participation continued
Dan killed himself six months into study participation
Postscript:
(Civil legal ugliness I won’t get into…)
FDA review: Not clear that Dan died because of study participation; no action taken
Prosecutorial review: Not clear that Dan died because of study participation; no criminal liability found; no action taken
Questions:
Would his drug regimen have been changed if he were not in a study?
Would his protections have been better if he had in fact been committed and had a guardian?
Was study participation in fact a contributor to his suicide (or was his schizophrenia beyond help)?
Postscript ii:
Several faculty members asked for an outside review, arguing that “nothing worth prosecuting” isn’t a high enough standard…
Third example:
• Popular anti-inflammatory drug about to have its patent expire
• It has an aspirin-like effect on platelets, so might prevent certain types of clots
• If so, it may make an intravenous form of the drug useful
• New formulation new patent $$$$
• Let’s try it!
The experiment worked:
• Intravenous drug prevented experimental clots in veins that had been chemically, thermally or abrasionally damaged
• Worked better than having the animals on warfarin for a week in advance of the injury
• Worked almost as well as heparin (the standard i.v. anticoagulant)
• The sponsor decided not to try the study in people
The experiment worked TOO WELL:
• The drug levels that prevented clots could be achieved with the oral form already on the market
• No new form no new patent no big bucks (Schillinge, actually – this was in Vienna before the Euro came in)
• When we now add this drug to a failing anticoagulant regimen, we’re doing so on the strength of unpublished bunny data
The three examples show several of the problems one might encounter in industry-sponsored research.As Claude Bernard warned 150 years ago,
going into research with a desired outcome in mind distorts the entire scientific process:
• What questions you ask• How you ask them• How you select subjects• How you analyze results• How (even whether) you report results
The three examples also suggest that the incentive structure might be important:If your desired outcome is support of product
development or marketing, it further influences:
• What questions you ask• How you ask them• How you select subjects• How you analyze results• How (even whether) you report results
How big a deal is this?
Marketing has become a much larger fraction of drug-company budgets;
Firewalls between marketing and research components of drug companies have largely been taken down;
This may now be a bigger concern than it was 25 years ago Carl Elliott, M.D.
The questions you ask:
• Questions that will likely gain evidence in support of your desired outcome (“counterproof”)
• Questions that will support the licensure and marketing of your product
• Questions that will demonstrate a niche in which your product might outperform others
The questions you might not ask:
• Questions that will critically challenge the hypotheses that underpin your desired outcome (“counterproof”)
• Questions that will sensitively detect the limitations of your product (toxicities; drug interactions)
• Questions that will demonstrate a niche in which your product might be outperformed by others
• Questions supporting strategies that do not lead to profit
How you ask the question:
• Choose subjects who are most likely to benefit; least likely to have problems
• Structure the competitor arms so that they are unlikely to be as good as they could be
• Choose a sample size and study interval that make it unlikely that you will detect/document any uncommon problems
• Make the primary endpoint something that advantages your product
Reality check:
• It’s not automatically evil to do a study that evaluates a product under the most favorable circumstances … you have to find out if it CAN work
• It becomes a problem when those studies are presented as though they are more representative and more definitive
• It becomes a problem when the more definitive and more generalizable studies never happen, or never get published
Reality check ii:
• It’s not just industry-sponsored research• The same sort of intellectual traps and
adverse incentive structures are encountered when someone is pursuing an idea on which he’s built his career, or when he’s pursuing his next major grant
How (or whether) you report results:
• Shout from the rooftops when you hypothesis (or product) is supported
• Publish in a high-profile journal• Distribute zillions of reprints
• Put the caveats in a separate publication with a lower profile
• Publish unfavorable studies obscurely or not at all
Is this a theoretical concern or real?
Let’s look at the CAFÉ study, the one that Dan was enrolled in:
• Authors included 3 AstraZenica employees and seven extramural psychiatrists
• 400 subjects• Primary endpoint: Still taking drug at one
year? (A “combined endpoint” for efficacy and tolerance)
• Non-inferiority design
CAFÉ study
• Only 119 of the 400 subjects were still taking the drug at one year
• Publication doesn’t tell why those who stopped the drug did so (side effects vs. ineffectiveness)
• Conclusion: “Non-inferior”
CAFÉ study
• With three groups and only 119 outcomes, a huge difference could fail to be detected statistically.
• For example, if the three groups had 25, 47 and 47 (rather than equal numbers of favorable outcomes), that would not have reached the customary statistical significance threshold of p < 0.05.
CAFÉ study• The study was underpowered to detect a
difference, but its primary outcome was “no difference.”
• That “no difference” is then described with phrases like “comparable efficacy,” obscuring the fact that the study was pretty insensitive.
• This turned out to be really important, because an unpublished study suggested that the sponsor’s drug might actually be inferior!
CAFÉ study
• No comparisons were made to older antipsychotic agents, such as haloperidol
• With the newer agents’ superiority being challenged, even a good study comparing the three newcomers might have been asking the wrong basic question.
• Carl Elliott (and others) asks: Was this study a marketing exercise or a scientific one?
Carl Elliott (reviewing several critiques of the CAFÉ study):
“If these experts are right, then the study in which Dan Markingson committed suicide was not simply a matter of inadequate informed consent, or financial conflicts of interest, or even failure to monitor a subject's care. The ethical breach was built into the study from the start. It is one thing to ask people to take risks for science, or the common good, or to help other people. It is another thing entirely to ask them to risk their lives for the marketing goals of AstraZeneca.”
One Study, or a Pattern?
Meta-analyses have been done both for newer antidepressants and for newer antipsychotic medications
Invariably, more than 3/4 of the published studies have reported results favoring the product of the sponsoring pharmaceutical company
This appears NOT just to be publication bias; the meta-analysts in some studies have looked for – and found – study design bias in a substantial proportion of reports
So Where Are WE?
The balance is tricky:
Pharmaceutical and device manufacturers have a need to test their products
Proof-of-concept studies (CAN it work?) have a legitimate place in product development – but often are limited in their generalizability
Real-world studies (will it work in practice?) may be more practical, but may miss niche utility
Studies comparing products may be very useful, but may also be misleading if designed (or promulgated) from a marketing perspective.
The balance is tricky:
Concealing study results that do not favor your product is wrong if it puts anybody at risk
Potential subjects have a right to know if the risk they are being asked to bear is to answer a marketing question, a curiosity question, or a question with immediate implications for best clinical practice
An IRB is charged to make the call whether the benefit of a study is enough to warrant the risks subjects are asked to bear -- who’s to help them decide the tough cases?
And just to make sure you’re challenged:
The Markingson case led to some lawsuits.One finding might surprise you: The drug
manufacturer’s fiduciary responsibility is to its shareholdersNo legal (as opposed to ethical) responsibility to put the welfare of the subjects ahead of that of the shareholders
Thus, no grounds for tort action if regulations complied with
If we could turn back the clock:
Direct marketing of drugs and devices to consumers has led to huge marketing juggernauts in many companies; the old firewalls between marketing and science were probably a good thing.
We can’t turn back the clock, but perhaps a regulatory structure could be devised to make it harder for marketing considerations to distort the science.
What we can do now:
We can be more diligent in trying to identify studies that are likely to be market driven, and make certain that their real nature is frankly disclosed to subjects.
We can support proposals that would require product-comparison studies to be independently designed and conducted, rather than being the property of someone with a stake in the outcome.
It’s not just the sponsors:
Comprehensive Cancer Centers are scored on the proportion of patients entered on study protocols – that creates a pressure to have a study (ANY study) for every patient
My own division has come under pressure to implement more Phase-I studies, because of their favorable impact on hospital and clinic revenue streams.
Claude Bernard saw it coming
But I’ll bet he didn’t foreseethe enormity of the financial
traps that are there toaugment the intellectual
ones. His warnings arejust as timely now as
in 1865 … but theirchallenge has grown.