STANFORD UNIVERSITY MEDICAL CENTER

STANFORD,

CALIFORNIA 94305 (415) 497-6970

RECEIVEDJUN 18 1982

The MYCIN Project

Diilsinn of

General

InternalMedicineDepartment of Medicine June 17, 1982

E. A. FEIGENBAUMDr. Peter GreenwaldDirectorDivision of Resources, Centers, and Community ActivitiesNational Cancer Institute

'j

Building 31, Room 3A16Rockville PikeBethesda, Md. 20205

Dear Dr. Greenwald:

I would like to take this opportunity to bring to your attention aproblem recently encountered during the review of a grant applicationsubmitted to the extramural program.of the National Cancer Institute. Theproblem is a general one that exists in other institutes as well, but whichappears to have particular relevance to current cancer research because ofrecent growing interest in the applications of computing technology to themanagement of oncology patients. The review of our proposal serves as anexample of the issue I would like to bring to your attention.

BACKGROUND: MEDICAL COMPUTER SCIENCE AND THE NIH

In brief, there is no branch of the NIH that has a direct charge tosupport basic research in clinical applications of computer science. Allmedical computer science grant proposals therefore tend to be assigned forreview to the institute for which the domain of application is mostrelevant. Such applications are often reviewed on the basis of theirshort-term relevance to the field of application, frequently without anappreciation for the basic computer science questions being addressed. Theresult has been recurring frustration among those of us who believe in thetremendous potential of computing technology for medicine. Despite ourbelief in the future of the field, we recognize that ideal systems are yearsaway because additional research is needed. The problems to be solved aresimilar to those in traditional "wet bench" basic research laboratories wherepayoffs are more generally accepted as being long term and incremental as newknowledge is acquired.

MEDICAL COMPUTING AND THE NATIONAL CANCER INSTITUTE

Although there are some computer-related programs supported by theNCI, we understand from our discussions with extramural staff that there areno formal guidelines either for the review of such proposals or for theirassignment to committees. For example., one such project is funded by theCooperative Group Clinical Trials Program and another by the Cancer ControlProgram. The lack of formal guidelines, however, leads to confusion indetermining the appropriateness of review committees (or a need for thecreation of special study sections), and reviewers themselves are apparently

*

Page 2Dr. Peter Greenwald

often uncertain about what criteria to use in assessing such proposals or indetermining their relevance to the programmatic interests of the NCI. To myknowledge, there is no review group with specific computer science expertise.When standing committees are selected for assignment of such applications, itis usually because they have biostatisticians or epidemiologists as members.

MY REASONS FOR WRITING

lam writing for two reasons: ( 1 ) to suggest that the 'NCl considerestablishing formal guidelines for the assignment and review of computer-related proposals (it may well be, for example, that site visits should berelatively routine for such applications and that special study sectionsincluding computer experts should be regularly employed), and (2) to requestthat a special study section be assigned to reconsider our proposal(1 RlB CA 31911) which was submitted in June 1981 and assigned to the CancerControl Grant Review Committee.

AN EXAMPLE: THE REVIEW OF OUR PROPOSAL

When the study section met to consider our proposal last October,they deferred a decision to await answers to several questions to which weresponded in December. We were concerned upon receiving the questions,however, because they seemed to suggest that the committee was failing toappreciate the research thrust of our proposal and was interpreting ourgoals, instead, as the development of a product for short-term clinicalapplication. With initial support from the National Library of Medicine, wehave developed a prototype chemotherapy consultation system using novelcomputer science techniques and advanced hardware technology. Our proposalcalled for further research to transfer the system to the next generation ofmachines which we believe will provide the optimal mechanism fordissemination of these kinds of clinical tools. At the next review committeemeeting in March of this year, our proposal was approved but with too low a

priority to be funded. We have had an opportunity to review the committee'ssummary statement and believe there is reason to question whether ourproposal was in fact reviewed by peers, especially in the area of medicalcomputer science. The orientation of the review committee is reflected instatements such as the following: "The primary weakness of this proposal is

that it is tied to a single computer system, and the generalizability andtransferability to community practice are not clear. Therefore, while thesystem could be very valuable in a sophisticated, university-based setting,it seems to have less relevance to community treatment settings." It isironic that this statement should occur precisely at a time when the clinicaloncology literature has included pleas for the development of techniques tobring clinical trials to the community setting (Seminars in Oncology,8:473-1477, 1981). The committee's failure to recognize the nature of theresearch we are proposing, and its failure to assess the effort as only afirst step toward longer-term transferability to community practice, reflectsa basic lack of understanding of medical computing research and the processwhereby new technology is developed and disseminated to the medical community

r3PageDr. Peter Greenwald

EVIDENCE FOR NCI INTEREST IN THE FIELD

It is ironic also that, as the Cancer Control Committee was reviewing

our proposal and questioning its relevance and utility, we were contacted by

Dr. Brenda Edwards of the Division of Cancer Treatment. She had been shown acopy of our proposal, was excited by the novel capabilities we are developing(and which differ from those in any of the other protocol management systems

under development around the country) , and asked us to participate at aWorkshop to be held at the NCI in July of this year. Apparently there arethose within the NCI who are extremely interested in the ways in which therapidly changing computer field may benefit the management of oancer patientsand the organization and analysis of clinical trial data. The lay press hasalso shown great interest in our work, and it was featured prominently in arecent issue of TIME Magazine (May 17, 1982, p. 71).

RESPONDING TO THE REVIEW

Our quandary at this point is a difficult one. We have spoken atlength both with Dr. Handelsman, Program Director for the ClinicalCooperative Groups, and with Dr. Browning, Executive Secretary for the CancerControl Grant Review Committee. Neither has been able to suggest an optimalway to respond to the committee's review nor to assure that the same kind ofreview won't occur if we resubmit. We believe in the work we have alreadyundertaken, know that it holds great promise, but are faced with a critique

that leaves us almost no specific criticisms to which we can respond in areapplication. The review committee's reservations did not focus on thescientific content of our proposal, but on their concerns regarding the shortterm utility of the developments described. There is little we can sayexcept that we disagree with that view of the field and believe that ourpeers who work in medical computing research, and specifically in oncologyapplications, would agree with our philosophy. Any new grant applicationwould necessarily look very similar to the one already reviewed. We believeit describes the only reasonable way to proceed with our research plan.

SUMMARY

In summary, although I would be willing to submit a new applicationwith some minor modifications that reflect the additional lessons learned inthe last ten months, I believe that the original grant application receivedsuboptimal review at the NCI, largely because it was not assessed by anadequate peer review group. Since this is a common problem for medicalcomputing applications, we had hoped we would be site-visited so that wemight try to educate the committee about the work and its potential.Unfortunately, the committee did not request such a visit, and we had nochoice but to try to explain our work to them in the application and in ourresponses to their questions. I would therefore like to request that you

4PageDr. Peter Greenwald

have our original proposal reconsidered, specifically by a special study

section. We are confident that a review group of peers will eitheracknowledge the quality of the work we propose to undertake or will provide

substantive comments to guide us constructively in rethinking the project and

rewriting an application. In the larger picture, however, the solution to

the more general problem that our situation represents will be the formation

of specific NCI guidelines for the review and assignment of computer-related

grant proposals.

Thank you for your attention to this matter, and I would be pleased

to discuss any aspect of this letter further if you wish.

Yours sincerely,

Edward H. Shortliffe, M.D.,Assistant Professor of Medicine

(General Internal Medicine) andof Computer Science

EHS/dv

cc: Dr. W. Raub, Assoc. Director for Extramural Research & Training, NIH

Dr. V. DeVita, Director, National Cancer Institute.✓Prof. E. Feigenbaum, Principal Investigator, SUMEX-AIM National Resource

Dr. H. Handelsman, Program Director, Clinical Cooperative Groups

Dr. R. Browning, Executive Secretary, Cancer Control Grant Committee

Dr. C. Jacobs, Co-Investigator, The ONCOCIN Project