Institutions of Objectivity in Clinical CancerTrials: A History and Sociology of Data

Monitoring Committees in Clinical Cancer Trials

Peter Keating & Alberto Cambrosio

Paper presented 03-12-08

A word from our sponsors: CIHR, FQRSC and SSHRC funded research for this presentation.

QuickTime™ and a decompressor

are needed to see this picture.

I. The emergence and development of DMCS

what are DMCs?• DMCs consist of group of experts who periodically review accumulating clinical

trial data for the purpose of monitoring safety and treatment efficacy

• After review, DMCs issue a stop or go judgment

• DMCs have raised epistemic and governance questions

‘There is irony in that, in the case of trials where we experiment on human beings, those who do so cannot be ‘trusted’ to see the data they generate while they so experiment. Surely clinical trials must be one of the few fields of scientific inquiry in an open society where researchers are so regarded’. [Meinert 1998a: 542]

‘Imagine a scientific methodology that bases its experimental technique on a technology that alters sample size, modifies the study design, and provides guidance to the interpretation of data, but which the methods sections of papers rarely mention and to which papers rarely allude at all. … In fact, participation on a data monitoring committee requires a vow of silence, a silence that often remains unbroken for years’. [Wittes 1993: 419]

DMCs in the United States: an overview

• less than 50% of clinical trials have a DMC

• 100% of NIH-sponsored multi-institutional trials have a DMC

• 42% of industry-sponsored trials have DMCs

• 90% of trials with DMCs are multi-centered

• most DMCs involve Phase II and Phase III trials

• trials with DMCs are stopped a significantly higher number of times than trials without DMCs

DMCs are deployed extensively in clinical cancer trials

participant history 1: the administrative origin

DMCs emerged in the late 1960s because of an administrative innovation at the National Heart Institute

• the 1967 Greenberg Report created the NIH model for clinical trials (first published in 1988: Controlled Clincial Trials)

• used in a number of clinical trials in the 1970s (but not in cancer)

• the NIH model is the source of present-day DMCs

the original Greenberg diagram

a recent representation of the NIH model

The Executive Committee has “the responsibility … to review the data analysis coming from the Coordinating Center and to prepare frequent reports to the participants, as well as annual reports to the National Heart Institute”

participant history 2: the statistician’s view

DMCs emerged as the result of innovations in the field of statistics

Armitage and McPherson (NEJM 1973)

• in 1969 Armitage and McPherson observed that repeated significance testing would lead to the observation of false positive results; they thus invented the first ‘stopping rules’ (Armitage, McPherson and Rowe, J.Roy. Stat. Soc., 1969)

• nonetheless, informal rules of thumb already functioned as ‘stopping rules’ in the 1970s

‘In the early days of ECOG ... we would look at the data formally every six months because we prepared … the agendas for a group. … So we were looking every six months, but we knew even as the theory was just developing, every statistician, every trained statistician knows that if you look too often you might stop a trial for which there is no difference; you stop it by chance alone. … And those were rules of thumb, quite useful rules and there was a very early paper by Haybittle [1971] that made that a bit more formal, that said you know, any decent rule of thumb in sequential design says: “look as often as you want”, well, not exactly but: “look fairly often, and only stop the trial or think about stopping the trial if you see a difference that’s more than 3 standard deviations above the null, or below the null”’. [Interview with David Harrington, Boston, 22 April 2005]

• in the mid 1970s Stuart Pocock, an ECOG statistician, generated the first group sequential designs for clinical trials

• further innovations in the early 1980s in group sequential methods made stopping rules useful for large trials

Sequential (2X2) medical trial:

Hodgkin’s Disease, Stanford Medical Center,

1967

The group methods

replaced the 2X2 sequential methods first developped and used in the

1960s

• a simple calculation: if during interim analysis the trial crosses a preset boundary then it must be stopped

• BUT: ‘the process process of making such judgments is rarely straightforward; the observation of a low p-value or a sequential boundary crossed represents only the beginning of a process, not the end’. [Ellenberger et al. 2002]

• the statistical procedures developed to monitor trials must be interpreted to produce decisions

• results of a trial must be ‘statistically’ convincing in order to stop a trial for reasons of ‘early success’ ... but convincing to whom: researchers or practitioners?

statistics and the ‘mechanical’ mandate of DMCs

‘I was involved in a trial in which there were no boundaries set in advance and the co-ordinating centre said that if we had used Pocock stopping rules we would have just barely crossed, if we used O’Brien-Fleming stopping rules we wouldn’t have, if we had used stochastic curtailing... and it became completely meaningless’. [Michael Proschan, 1994]

• In the mid 1990s, the proliferation of rules sometimes raised issues of choice: which rules should apply?

what do the participant histories omit?

• between the late 1960s and the advent of DMCs (in cancer) in the mid 1980s, clinical trials changed in scale and scope to become a new ‘style of practice’

• this transformation entailed the emergence of full-time clinical cancer trialists interested in the success and objectivity of trialsASCO Membership and Annual Meeting Attendance

0

5000

10000

15000

20000

25000

30000

1964 1968 1972 1976 1980 1984 1988 1992 1996 2000

Year

Mem

bers

hip

/Att

en

dan

ce

Membership

Annual Meeting Attendance

cancer clinical trials: a new style of practice

Style of practice = distinctive configuration of institutions, epistemic concerns and materials that generates specific ways of identifying and investigating research questions, of producing and assessing results, and of regulating these activities

• observational sciences• field sciences• laboratory sciences• statistical sciences• clinical sciences• new sentences• new evidence

[Hacking]

• new classes• new criteria• new objects• new explanations•new intersubjectivity

the elements of style in clinical cancer reseach

• a continuum of activities: from screening to treatment through research, testing and training

• performed by new collective entities, the cooperative oncology groups

• a distributed, multi-center network

• held together AND performed by and through protocols

• entrenched in new institutions: data centers, data managers, DMCs, etc.

The Groups are often internatio

nal in scope

SWOG-S0106 includes 17 sites in Canada and 8 sites in Sweden

QuickTime™ and a decompressor

are needed to see this picture.

the data center

• Establishment of centralized data centers in 1970 in the US (ECOG), and in 1974 in the EU (EORTC)• EORTC (1974): 1 coordinator, 2 statisticians, 2 computer analysts, 4

“data managers” (a new term), 2 secretaries, 2 US consultants

• Involvement in the cooperative groups’ activities from the planning stages to the final report of a study’s results:• Protocol design, estimate of number of patients required and expected

study duration• Design of study forms: standardization of data forms, structure of data

files and routine procedures for data handling•‘Designing forms for data collection in clinical trials: it was a big advance. It is hard to imagine that a pedestrian thing like that would be a big advance’ (Zelen)

• Central registration and randomization of patients• Review of received patient forms for ambiguities, completeness or

protocol violation• Presentation of interim administrative reports (including summaries of

any toxicities observed) and statistical analyses at regular meetings with the cooperative group

• Detailed statistical analysis at the end of the trial•‘We applied all kinds of sophisticated statistical techniques for data analysis. Before, the statistical techniques were very pedestrian. In fact, we developed a lot of statistical software that no one else had’ (Zelen)

The EORTC Data

Center: An instance of ‘Central Command’

CALGB 2006

The protocol lies at the core of the new style of practice

QuickTime™ and a decompressor

are needed to see this picture.

DMCs as a solution to the problem of too much

information and too few patients

• the cooperative groups discovered that the circulation of information about positive or negative results hindered participation:

• North Central Cancer Treatment Group inaugurated the first DMC in 1977

• SWOG followed suit in 1984, ECOG in 1990

• in 1992 the NCI made DMCs mandatory for all cooperative groups

DMCs were thus established in order to:

• restrict data circulation during long-term clinical trials: the original impetus was not to protect patients

• reduce apparent and real conflicts of interests

• protect patients

• assure good governance

‘Because clinical trials are under increasing public scrutiny, we must use procedures that protect our research against the appearance of impropriety’ [R. Simon and R. Ungerleider, NCI, 1997]

the road since 1992

• 1994: NCI instituted policy requiring DMCs to be independent from cooperative group investigators

• Statisticians from seven major cooperative groups initially strongly opposed this policy

‘Monitoring a clinical trial is a complex task, best conducted by a small group integrating all the issues that must be addressed. This group should freely and confidentially discuss patient safety, the accuracy of accrual and outcome predictions made at the design stage, protocol compliance, unexpected side effects, recent results from other, similar trials, the place of the trial in the scientific program of the organization conducting the study, and, when appropriate, outcome data. These issues are not separable and should not be discussed in isolation’. [Statisticians for seven major cooperative groups]

• Opposition subsequently subsided

‘I watched [the independent DMC] grow from a group of ten or twelve people who really didn’t know about the trials they were monitoring to a group that worked very hard to understand the context of those trials in cancer research. … I didn’t see the conflict of interest issue being as large as the NCI people did. I still don’t but I have seen the external data monitoring committees work reasonably well and have even been on a few of them, so in that sense I think it’s turned out to be a good thing’. [Interview with David Harrington, Boston, 25 April 2005]

II. What kind of institution is a DMC? DMCs are reflexive

institutions.

reflexivity as a form of endogenous critical inquiry• ‘endogenous critical inquiry’ (Lynch 1982):

‘practically solves the issue of what counts as the relevant contingent relationship between “objective” and “socio-historical’ states of affairs’

• the search for an objective assessment of interim data, far from resting solely on methodological principles, resorts to reflexive practices focusing on:

• the qualifications of DMC members

• their attributes and prerogatives

• the extent to which DMCs should be able to redefine the parameters of a trial and restrict the circulation of data

• In sum: DMCs institutionalize endogenous critical inquiry

A typical DMC charter

Informal reflexion and formal coding

The Guidance process

i) 2001 Draft Guidance document issued

ii) 2001 Public Meeting

iii) 2006 Guidance document issued

Forums for Reflexion

(1) role and purposes of DMCs

• should a DMC have the power to change a trial protocol and under what conditions?

‘In at least three DMCs that I’ve been a part of, rather acrimonious fights erupted at the beginning about my raising these kinds of issues, charges being made that this is a data monitoring committee; those are IRB or steering committee functions; it’s not for the DMC to do’. [Norman Fost, FDA meeting]

should a DMC be allowed to change sample size?

‘A clinical trial ... is not a fixed quantity. It’s almost like a living thing. It evolves; it changes; it can change. One of the obvious ways in which a clinical trial might need to be modified has to do with the sample size. When the sample size is calculated, different things are assumed about the rate in the control arm, the rate in the treatment arm. Those assumptions may or may not be valid. And it may turn out that the trial is underpowered and the sample size needs to be adjusted. A data monitoring committee, although it’s not easy, can grapple with this’. [Gregory Campbell, FDA meeting]

should a DMC be allowed to change sample size?

‘[a DMC] is seeing data on efficacy and for it to have the ability and the right to change end points and to change crucial aspects of design I think can sacrifice the integrity of the design’. [Janet Wittes, FDA meeting]

‘the unblinded data monitoring committee really can’t credibly change end point, sample size, subset plans or anything, any more than an unblinded sponsor could without at a minimum affecting alpha or introducing bias that we don’t know how to correct’. [Robert Temple, FDA meeting]

should a DMC be allowed to change sample size?

‘Not all studies are confirmatory but those studies that are confirmatory, I’d like to be able to interpret them in that manner. It means ... I’d like to have a pre-specified hypothesis that I then confirm. At the same time, there can well be during the course of a long trial external information that could enlighten us as to what the hypothesis really ought to be. I actually don’t have a problem if I’m certain that it’s external data that leads to refinement and this is the essence of where this independence and separation enables or empowers the sponsor to have that flexibility’. [Thomas Fleming, FDA meeting]

at what point does monitoring become intervention?

• DMCs can be asked to validate initial designs and to participate in final analysis

• should DMC members sign papers produced by these trials?

• ‘adaptive designs’ allow DMCs to redesign the trial on the basis of interim results.

• if DMCs participate in the design and/or redesign:

• do they have interests in the trial?

• are they still objective?

(2) structure and membership of DMCs

• in Britain, DMCs organized by the MRC have 3 members: two clinicians and a statistician; in the United States, DMCs may have five times as many members

• what kind of expertise is needed for a DMC to function properly (‘objectively’)?

• how to address potential conflicts of interest?

• what kind of decision do DMCs seek: i) expert diagnosis, ii) in-house consensus or iii) majority rule by constituency ‘representatives’?

‘I’ve often had it said that we have to have an odd number of people on the DMC so that when we vote it won’t come out tied. I object strongly to votes on DMCs. I believe that the DMC’s responsibility should include discussing issues at a length and in a depth to arrive at consensus about what

ought to be done’. [Thomas Fleming, FDA meeting]

the issue of interests

• DMC members may have financial and research interests

• clinicians as consultants to pharmaceutical and biotech companies

• program interests of government sponsors

• ‘disease interests’ of patient representatives

• and yet, members without interests are not interesting

• endogenous critical inquiry focuses on the search for ‘acceptable levels of conflict of interest’

how many statisticians?

• DMCs should be provided with three statisticians:

• a statistician member of the trials’ protocol team or steering committee: BLINDED to the data

• a statistician reporting to the DMC but independent of the trial sponsor: UNBLINDED access to the data

• a statutory statistician member of the DMC

‘Some wag said that this document is a full employment act for statisticians’. [DeMets, FDA Meeting]

The problem: ‘Suppose that a trial steering committee were considering changing the primary endpoint of the trial. The statistician, knowing the interim results to that point, would know whether the drug being tested was in fact showing a stronger effect on the ‘new’ endpoint than the original endpoint, and would therefore be in an awkward position during this discussion. If the change were to be made, and if final analysis showed a significant effect on the new endpoint, there could be the perception that the interim data influenced the change – that if the trend for treatment effect had been different, the statistician would somehow have found a way to discourage the change in endpoint’.

Does the independent staistician have the expertise?

‘In the multicenter, cancer cooperative group setting … the trial data manager prepares the data for the interim analysis which is carried out by the trial statistician. Within this setting we feel that it is difficult for a completely independent third party to carry out the analysis for the following reasons: 1. They will not be as familiar with the trial or the data as the trial data manager and trial statistician and are thus more likely to make mistakes in the analysis. 2. With a large number of trials it becomes much more difficult both practically and financially to have the trial analyzed by a third party’. [Richard Sylvester, FDA Dockets]

Should the committee itself remain blind?

• DMCs established to keep interim results confidential so as not to compromise completion of the trial

• what results, if any, should be transferred to those conducting the trial?

• should DMCs monitoring different but complementary trials be allowed to share information?

• how can DMC members themselves maintain clinical equipoise if they know that the interim results are positive or negative?

Is there a DMC culture and know-how?‘[DMC] decisions and the discussions are very complicated, they’re very nuanced, and they reflect a certain sociology of a committee that varies from committee to committee ... go to every DMC … that you can go to because you can learn a lot, it’s the only way you’re going to understand it and it’s really fun’. [Wittes, FDA Meeting]

‘the need for folks specifically experienced and mentored in the process of DMC activities is really very critical and the confidence with which folks are able to invest the responsibilities into the groups is very importantly based on the talent base that exists to accomplish those goals’. [Conner, FDA Meeting]

create an ‘apprentice’ system to pair inexperienced statisticians with experienced statisticians on DMCs?

‘Since there are no accepted standards of what constitutes conclusive evidence, the process by which (DMC members) arrive at their decisions must be made transparent’ [NEJM 2004]

III. DMCs and objectivity

• objectivity as a historical construct:

•regimes of objectivity (Daston & Galison)•true-to-nature •mechanical (a-perspectival) objectivity• trained judgment

a new form of objectivity for biomedicine

True to nature: selection and synthesis

Mechanical objectivity: automatic transfer

Trained judgment: pattern recognition

a new form of objectivity for biomedicine

•regulatory objectivity: less an epistemic virtue than a product of institutional action

•generates and institutes conventions through concerted programs of collective action based on the use of a variety of systems for the production of evidence (e.g., clinical trials), that in turn call upon standard substances and standard practices (e.g. practice guidelines)

•incorporates unprecedented levels of reflexivity: in their debates and discussions, biomedical practitioners take into account the conventional dimension of their endeavors

•corresponds to a new regimen of coordination of medical practices that increasingly operates as a condition of possibility for the very existence of these practices: regulatory mechanisms are endogenous to and constitutive of research practices

DMCs and regulatory objectivity

• DMCs have become part of the design of large-scale cancer clinical trials within which they operate as an institutionalized form of endogenous inquiry.

• The implication of multiple agencies and institutions that seek the governance of clinical trials through objectivity and the reflexive layering and interlocking of regulatory arrangements serve to generate regulatory objectivity.

• The kind of objectivity produced by DMCS is not reducible to an absence of ‘subjective bias’ or ‘conflict of interests’: the ‘integrity’ of the clinical trial process, established by DMCs, is also a sought-after good.

Following the rules

• DMCs can be seen as a solution to the wittgensteinian problem of rules: every rule requires an interpretation, every interpretation, a rule for interpretation.

• BUT: DMCs are not a sociological solution to the problem of infinite regress: acculturation to a specific community (i.e ‘this is how we do it’).

• Instead: the rules are constantly revised in light of contradictory and alternative interpretations and emergent values.