Expert Judgement in Pharmacoeconomic Studies

Expert Judgement inPharmacoeconomic StudiesGuidance and Future Use

Christopher Evans and Bruce CrawfordMAPI Values USA, Boston, Massachusetts, USA

Abstract Research in the field of pharmacoeconomics has increased substantially dur-ing the past decade. Much of this research has been on the design and analysis ofdata concerning the relative merits of one drug or device compared with anotherin terms of costs and effects. Concomitant with these evaluations has been thedevelopment of guidelines for the conduct of economic evaluations in severalcountries. However, despite an increase in research, little attention has been paidto how different study designs may influence the results of a study. The use ofexpert judgement in decision analytic modelling is one area where design issuesmay influence the findings of a study. This issue is examined for the case ofmodified Delphi and Delphi panels.Although the use of expert opinion in modelling studies seems to be wide-

spread, there is little consistent application, understanding or reporting of thetechniques used. In particular, the definitions of techniques vary between studies,the criteria for determining when consensus is reached vary, and the reporting ofthese criteria is absent. Future studies using expert judgement should be moreaware of the controversies surrounding the issue and provide more reporting ofthe techniques used. It is proposed that future validation exercises may assistresearchers in determining the most appropriate application of methods.

CURRENT OPINION Pharmacoeconomics 2000 Jun; 17 (6): 545-5531170-7690/00/0006-0545/$20.00/0

© Adis International Limited. All rights reserved.

In many retrospective pharmacoeconomic eval-uations, recourse to a panel of experts is made inorder to obtain information that is not availablefrom other sources, such as clinical trial data, hos-pital and GP records and administrative and epi-demiological databases. Expert opinion is foundmost often in modelling studies; however, it mayalso be used in the context of pharmacoeconomicstudies conducted alongside randomised control-led clinical trials. The use of expert opinion offersseveral advantages compared with other data col-lection techniques: it is not an expensive or time-consuming activity and it offers an additional sourceof data when patient records do not contain com-

plete information. Expert opinion has been used inpharmacoeconomic studies to collect informationon transition probabilities,[1] resource utilisation,[1]costs,[1] and initial model structure,[1] to validatethe face and content validity of a particular model,to remove protocol-induced costs and to converttreatment patterns from one country to another.[1-3]The general rationale for the use of expert opin-

ion in pharmacoeconomic studies is that: (i) it is anecessary component of research –without the datait would be impossible or prohibitively expensiveto examine some research questions; (ii) expertsare assumed to provide reasonable estimates of theitems under question; and (iii) the use of an iterative

consensus-generating process may improve thoseestimates so that they converge on the value thatwould have been obtained had there been enoughmoney to conduct a prospective trial.Further, if the techniques are used sparingly, the

overall evaluation will suffer little loss in internalvalidity and bias will not be a problem. For in-stance, experts may be used to derive informationon resource utilisation for a model. If the experts’estimates for the resource utilisation are slightlyinaccurate, little damage is done to the model: thedriver for a model is typically the estimate of theprobability of having an event with an associatedresource use and cost. If the probability estimate isobtained from trial data, there is little risk that thefinal estimate will be biased even when the re-source use is derived from experts (although it maynot be precise). Of course, the use of a sensitivityanalysis around the estimated parameter will offeran idea of the robustness of the estimate.Prior research in the area of expert opinion in

pharmacoeconomic studies has indicated thatthere is little consistency in the use of this source ofdata.[1] Some researchers use the term Delphi ormodified Delphi panel with a broad definition.Most researchers use panel data to collect infor-mation on resource utilisation; however, some alsoderive probability estimates. The selection and at-trition of panellists is another area of concern.As the use of expert opinion is widespread in

retrospective evaluations and is generally em-ployed without appropriate concern for its poten-tial weaknesses, an examination of the controver-sies surrounding the use of expert opinion is war-ranted. This paper identifies how existing guide-lines deal with expert opinion, discusses some ofthe more contentious areas and provides some rec-ommendations for the use of expert opinion. It isrecognised that there are a variety of opinion elic-itation techniques that can be used in pharmaco-economic evaluations to generate information onresource use, costs and probabilities: nominal grouptechniques, focus groups and various consensus-generating processes.[4-6] The discussion presentedhere concentrates on 2 types of techniques: Delphi

and ‘modified’Delphi panels. The focus is on thesetwo as they are believed to be the most common ineconomic evaluations and many of the design ele-ments overlap with other methods. This paper con-cludes with recommendations on how to ensurethat expert opinion plays an appropriate role inpharmacoeconomic analyses.

1. Guidelines and Recommendations

Pharmacoeconomic research is regulated by aseries of formal and informal guidelines and therecommendations by professional societies, academ-ics and consultants. In general, where guidelinesare promulgated, the tendency is to provide counselas to the information on what should be includedin an economic evaluation rather than how to col-lect data. For the use of expert panel data, guide-lines and the opinion of researchers suggest that thedata may prove useful in certain situations.The Australian Pharmaceutical Benefits Advi-

sory Committee guidelines[2] permit studies that useexpert opinion. The guidelines note that expert opin-ion is not a substitute for sound scientific evidenceand that data from experts will only be consideredin the absence of information from randomised andnonrandomised studies. The guidelines do not spec-ify every situation where expert judgement may beused. However, it is noted that expert opinion maybe useful in defining the place of the drug in pro-posed treatment, to modify resource utilisation pat-terns found in non–Australian-based studies, to ex-clude protocol-driven costs, and to predict resourceutilisation that was not followed-up in a clinicaltrial.The Canadian guidelines[3] on the economic

evaluation of pharmaceuticals neither endorse norreject the use of expert judgement in pharmaco-economic studies. The guidelines make mention ofpanel data as a source for parameters and refer tothe Guidance Document for the Costing Process.[7]This report suggests that Delphi panels may be use-ful in justifying treatment patterns and resourceutilisation that were reported for other countriesand determining if these patterns are similar to thatfound in Canada. This document suggests that this

546 Evans & Crawford

© Adis International Limited. All rights reserved. Pharmacoeconomics 2000 Jun; 17 (6)

is a possible solution, albeit probably unsatisfac-tory. Canadian researchers[8] have estimated thescale of the use of expert data in Canadian evalua-tions. Twelve studies commissioned by the Cana-dian Coordinating Office for Health TechnologyAssessment were reviewed and it was determinedthat one-third of the studies under this programmeutilised expert judgement to collect information onresource utilisation.The Dutch guidelines[9] do not provide any spe-

cific recommendation regarding the use of expertjudgement. Guideline 19, which refers to model-ling practices, notes that modelling studies may beused in pharmacoeconomic evaluations for trans-lation of efficacy data to effectiveness data and toadjust treatment patterns between countries. How-ever, as far as the guideline is concerned the bestmethod for doing this has not been identified.Although the US does not have any formal

guidelines for the conduct and presentation of phar-macoeconomic information, a multidisciplinarypanel appointed by the US Public Health Servicehas provided recommendations.[10] The advice ofthis group is that expert judgement should be usedprudently for probability assessments and whereno other sources of data are available. Where datafrom experts is felt to be necessary in the conductof economic evaluations, structured elicitation tech-niques are recommended. As is consistent with theAustralian and Canadian formal recommendations,expert estimates should be subject to sensitivityanalyses.Several researchers have suggested that expert

judgement can be successfully employed in phar-macoeconomic studies.Most, if not all, researchersacknowledge that expert opinion should be used asa last resort in pharmacoeconomic studies. Barr andSchumacher[11] support its use when ideal data arenot available and when, together with informationfrom meta-analyses and other trial data, expert opin-ions will serve as reasonable approximations. Nuij-ten et al.[12] also acknowledge the weaknesses in-herent in the use of expert opinion; however, itsapplication is not proscribed in modelling studies.Evans[1] suggests that the use of expert opinion

need not be avoided as long as the potentialweaknesses are addressed and the techniques areemployed appropriately. Similarly, Halpern et al.[13]recognise that expert opinion plays an importantrole in modelling studies, but is subject to severalerrors and biases.

2. Issues in the Use of Expert Opinion

Problems with the use of expert opinion ariseboth in terms of the accuracy of the estimates aswell as the generalisability of the estimates. Sev-eral troublesome areas are evident: (i) the provi-sion of seed algorithms to experts; (ii) the attritionrate of panellists; (iii) the qualification of experts;(iv) the definition of consensus; (v) the use ofterms; and (vi) the application and reporting of re-sults. How researchers have handled these issueshas been reported previously.[1] However, 3 issuescontinue to cause problems in pharmacoeconomicresearch: the inconsistent use of terms, the defini-tion of consensus and the application and reportingof results.

2.1 Inconsistent Use of Terms

The Delphi technique was developed by re-searchers at the RAND Corporation who were in-terested in the use of expert opinion in forecasting.The purpose of the Delphi method, at its initialdevelopment, was ‘to obtain the most reliable con-sensus of opinion of a group of experts. It attemptsto do this by a series of intensive questionnairesinterspersed with controlled feedback.’[14] Underthis process, several rounds of anonymous infor-mation gathering would occur until consensus wasreached.The inconsistent use of terms to describe the

expert opinion generating process is most commonwith the use of the ‘modified Delphi’ panel. Thisterm is used with a great deal of elasticity and itsdefinition changes from study to study. Frequently,the term is used to suggest that there is a singleconcept known as a ‘modified Delphi’ panel ratherthan being one of many types of a modification toa Delphi technique. For instance, a modified Del-phi panel has been used by researchers[1] to denote

Expert Judgement in Pharmacoeconomics 547


a panel that does not go through the full iterativeprocess but is terminated at a prespecified point(usually after 2 rounds). Halpern et al.[13] use theterm modified Delphi panel to denote a processwhere information derived from a panel is subjectto review by outside experts. In some instances inpharmacoeconomic research, it is unclear as to howthe Delphi panel was altered.[15]This lack of consistency in the use of the terms

makes it difficult to identify the potential impactthat the elicitation technique has had on the preci-sion of results in a study. For instance, if the studyis identified as a Delphi process when in fact it isa simple panel, the benefits of the iterative processare lost. These benefits may be substantial, as re-search has shown that experts are willing to correctinitial responses when faced with the anonymousresponses of other experts.[16]

2.2 Definition of Consensus

As the ultimate goal of a Delphi process is toreach a consensus of opinion, how consensus isdefined is important. Consensus in the context of aDelphi process need not equal exact agreementamong respondents. Consensus in the Delphi pro-cess may be assumed to have been reached when acertain percentage of the responses fall within aprescribed range.[17] As the Delphi method relieson an iterative process it is also important to exam-ine the degree of convergence or how much moreagreement is achieved in succeeding rounds com-pared to the first round.[18] Related to these issuesof convergence and consensus is the stability of theestimates. Stability refers to the degree to which in-dividual and group responses alter between rounds.Aprevious review of the literature indicated that

the use of expert opinion in pharmacoeconomic re-search did not rely on the Delphi method but on amodification of it.[1] The implications of these ap-plications is that consensus is being arbitrarilydefined by researchers prior to consensus beingreached as defined by an objective criteria.Guidance for determining when consensus and

stability are reached in the context of a pharma-coeconomic study has not been developed. Pre-

vious research has focused on determining whenstability may be reached in the context of ratingscales.[17] When scales are used, a 15% change levelaround the mode was used to represent a state ofequilibrium between rounds. Under this condition,changes less than 15% are viewed as stable andchanges greater than 15% for questions betweenrounds indicate that the Delphi process should con-tinue to additional round(s).[17]

2.3 Inconsistent Application and Reporting

There are no universal standards in the appli-cation and reporting of expert opinion in phar-macoeconomic studies. The absence of standardshas important implications for interpreting thevalidity of study results. An example from the areaof periodontal disease illustrates this point. Re-searchers at Harvard[19] conducted a hypotheticalcost-effectiveness model of alternative methods ofperiodontal disease control. As part of that model,experts provided probabilities of tooth loss. Eigh-teen periodontists from New England dental schoolswere administered personal interviews to providea variety of probability estimates. As part of thispaper, the authors reported that the results of theanalysis depended critically on the use of expertjudgement and that there were several potentialsources of bias (the subjective nature of probabilityassessment, lack of understanding of how expertsformed their estimates and the nonrepresentativesampling of experts).Researchers in Maryland and others[20] recently

constructed a cost-effectiveness model of 2 treat-ments of periodontal disease control. As part of thismodel a probability table was created that linkedprobing pocket depth and bleeding on probing tofuture treatment needs. Only 2 experts were chosenfor the probability estimation, both of whom wereinvolved in the clinical trial that was used in themodel. Although there was an acknowledgementthat the use of this type of data weakened the con-clusions of the model, the rationale for using only2 experts who had direct involvement in the devel-opment of the novel product was not substantiated.



3. Discussion

Five issues seem to be of critical concern todayin the use of expert opinion: (i) the selection ofthe appropriate type of group process and the cor-rect identification and definition for that process;(ii) the definition of consensus or stability; (iii) theuse of expert opinion; (iv) issues in the reportingof the use of experts; and (v) the potential for val-idation exercises.

3.1 Definition of Terms

A Delphi panel should conform to the originaldefinition used by the developers of the technique.Researchers should specify what a ‘modified Del-phi’ panel is. If a dominant definition of a ‘modi-fied Delphi’panel does not materialise, researchersshould refer to a modification to the Delphi paneland specify the modification made. This will makeit easier for reviewers and interested researchers tounderstand how the process was changed and toestimate the impact that the change may have onthe accuracy and the generalisability of estimates.Recently, several researchers have suggested a

framework for modelling practices. As part of thatframework they recommended using a modifiedDelphi approach ‘where information from expertopinion is reviewed by outside individuals andcomments from these outside reviewers are sub-sequently presented to the original experts. The ex-perts then have the opportunity to modify theiropinions.’[13] We do not prefer this definition, as itis unclear as to the usefulness of this approach. Ifit is thought that outside experts can improve thevalidity of the estimates they should be incorp-orated into the panel initially. If they are incorp-orated into the panel their estimates may be re-viewed and directly contribute to the final value.Our preferred minimum definition is one where

a modified Delphi panel has been altered so thatthe consensus process is arbitrarily stopped after 2rounds. This seems to be a practical solution to theuse of expert opinion in economic evaluations. Ithas the advantage of providing 1 round of responsefeedback to all members of the panel with the op-

portunity to change initial responses. In addition,it will minimise the problem of response fatigue bypanellists and it will be less costly and time con-suming to conduct than a full Delphi panel. Ourexperience with the technique shows that conver-gence occurs mainly after the first full iteration,which suggests that a 2-round process may be use-ful without compromising much in terms of theresults.Another researcher,[21] who is critical of theDelphi approach, has also noted that there are di-minishing returns associated with the third andfourth rounds so far as convergence of opinion isconcerned. However, it is important to rememberthat this technique has a serious limitation in thatthe appropriate convergence of opinion may nothave occurred by the end of round 2.On the whole, we see little reason to prefer a

simple panel of experts or focus group when col-lecting information for pharmacoeconomic stud-ies. As in the Delphi panel, there can also be dis-enchantment within the group. The lack of ano-nymity and the presence of a moderator have im-portant consequences. First, an individual maydominate the group so those more reserved mem-bers of the group do not contribute to the results ofthe panel. Secondly, the moderator of the panelmay unwittingly, or by design, bias the responsesof the group one way or another. Thirdly, the im-mediate nature of the focus group means that par-ticipants do not have a substantial amount of timeto reflect upon their answers and make possiblechanges.

3.2 Consensus

There is a lack of consistency and, indeed, a lackof reporting on the definition of consensus or sta-bility used in Delphi panels. Research conductedin other areas that have used Delphi panels may notbe transferable to the field of pharmacoeconomics.Convergence of opinion may be measured by ex-amining the average spread between quartiles ateach round. However, it is unclear at what point theaverage spread (presumably) decreases so far to im-ply that consensus has been achieved. If the meanis selected as the appropriate measure of central



tendency, the standard deviation may be examinedto determine the level of consensus: large standarddeviations would be associated with weak consen-sus and smaller ones with higher consensus. Thedetermination of stability is equally problematic.[17]Further research, perhaps as part of the valida-

tion studies mentioned in section 3.5, needs to beconducted so that an appropriate measure of consen-sus can be developed and recommendations aboutthe interpretation of that measure can be made. Un-til that occurs, researchers should be careful to pre-specify the measure and level, rather than wait untilthe end of a Delphi process to create post-hoc def-initions.Although the Delphi technique was originally

developed to generate consensus, it is also usefulto examine the results of Delphi panels to deter-mine areas of disagreement. For example, a Delphipanel that is used to collect information on resourceutilisation may generate bimodal distributions forresource use on the initial round. Although this typeof distribution may disappear in subsequent roundsas responses are altered, it would also suggest thatthere may be some underlying problem in the panelor that there are substantial treatment differences.For example, a bimodal distribution may suggestthat some of the participants selected for the panelare not expert in the field or are specialists. It mayalso be an indicator that the inclusion criteria forexpert selection were too general, which wouldlead to an over- or under-sampling of physiciansbased on experience, age or location. In addition, abimodal distribution may suggest that there arewidely divergent treatment patterns in a country,which would certainly have an impact on pharmaco-economic analyses.

3.3 The Use of Expert Opinion

The use of expert opinion in pharmacoeconomicstudies should be kept to a minimum. Expert opin-ion is currently used when there is no better sourceof data. The implication of this is that it is alwaysacceptable to use this source when nothing else isavailable. However, this may not be the case, andit is a distinct possibility that some pharmaco-

economic studies should be abandoned, or shouldnot have been conducted in the first instance, ifexpert opinion needs to be employed. A particular-ly troubling area is the use of expert opinion to de-rive probability estimates. Early research suggeststhat certain heuristics are often employed whenmaking judgements under uncertainty.[22] Theuse of these heuristics often leads to the presenceof systematic biases in data. It has been noted[11]that the occurrence of rare events may be overesti-mated and the frequency of common events may beunderestimated. Table I provides a ranking of theusefulness and credibility of expert judgementused in pharmacoeconomic studies.The use of a sensitivity analysis will only par-

tially correct for this problem. It is a well knowntenet in pharmacoeconomic research that estimatesthat are suspect or are considered important ‘driv-ers’of a model are subject to alteration through uni-or multivariate sensitivity analyses. In some in-stances, because of the original source of the data,a sensitivity analysis may fail to reveal any signif-icant information. For instance, if we assume thatthe mean probability estimate used in a model willbe systematically underestimated, then it is possi-ble that the standard error of that estimate will benarrow. If the range (based on the standard error)is too narrow, the sensitivity analysis may fail tocapture the true underlying value. If the range istoo wide, a model can be invalidated when in factit was an accurate representation of reality. Recentadvances in the use of sensitivity analyses in phar-macoeconomic studies may offer methodologicalimprovements in the calculation of relative and

Table I. Hierarchy of data derived from experts. Categories withhigher ranks reflect the fact that experts will, in general, providemore accurate and credible estimates

Rank CategoryI Validation of modelsIIa Treatment patterns specific to a settingIIb Treatment patterns in general for a countryIIIa Resource use specific to a settingIIIb Resource use for various settingsIV CostsV Probability estimates



overall sensitivity of variables; however, thesetechniques require additional validation.

3.4 Reporting

Additional reporting of the use and results ofexpert estimates is required in published pharma-coeconomic studies. It has previously been re-ported that it is often difficult to determine themodification made to Delphi panels and how con-sensus was reached.[1] Recently, Nuijten et al.,[23]as part of the Dutch guideline preparation activi-ties, supported a comprehensive disclosure of allaspects of expert opinion elicitation because of thepotential problem with bias. In addition, pharma-coeconomic reports should detail the methodologyselected and the processes followed.[12]Reporting of the use of expert opinion should

include the items listed in table II.

3.5 Validation and Reliability Testing

In the field of pharmacoeconomics, to our knowl-edge, there have been no formal attempts to assessthe validity of estimates derived from expert opin-ion. It is not necessarily an easy task to validatepanel data. The presence of panel data in a phar-macoeconomic study is a marker for the fact thatbetter data did not exist for the purposes of theresearch in the first place.There are 2 ways to validate panel data: (i)

through consistency of investigations, or (ii) a pro-spective validation study. Consistency of investi-gations refers to the use of different studies to gen-erate estimates of resource utilisation, costs or tran-sition probabilities and then compare these studiesin order to demonstrate similar or dissimilar re-sults. For instance, if a model is generated in thefield of acute treatment of ischaemic stroke, theestimates for the usual care arm of that model canbe compared to other prospective studies of acutestroke treatment that utilised standard care to de-termine if the final results are broadly compara-ble.[24] Of course, each study will differ in manyelements and time periods; however, if the overallresults are similar some confidence can be gainedthat the use of expert estimates was warranted.

A prospective validation study is a better methodof assessing the accuracy of expert estimates. Thistype of study can be conducted in 1 of 2 ways. Itcan either be conducted as part of an existing trialor as a stand-alone study.The first method would be to conduct the vali-

dation study at phase II in a particular therapeuticarea and use the results of that study to determineif the use of expert opinion is appropriate in laterphase research. The second method would be toconduct a ‘generic’ validation study. Under thisscenario, a validation study would be conducted inany therapeutic areas where ‘gold standard’ infor-mation exists and the information obtained wouldprovide a general statement as to whether a panelor (modified) Delphi process is a useful source ofinformation in pharmacoeconomic studies. Thus,for instance, if expert opinion is used in the devel-opment of a model in Parkinson’s disease it maybe worthwhile to do a validation study in anothertherapeutic area with gold standard data to ascer-tain the level of agreement between the differentdata sources. The findings of the validation studywould provide a general statement as to whether

Table II. Items to be included in reports of the use of expert opinion

How experts were identified (through the literature or through alist provided by a sponsoring company)Criteria for selecting experts, the number of experts at thebeginning and end of the study, and their qualifications andtreatment settingsWhether or not seed algorithms or literature reviews wereprovided to expertsIf literature reviews were included, the techniques used toidentify the relevant literatureHow the questionnaire was developedThe type of data collected from expertsThe type of elicitation technique usedIf a modification of the technique was made, the type ofmodification and the potential impact on study resultsThe number of rounds (if a Delphi panel) for the entirequestionnaire and per questionThe definition of consensus and stability and whether or not thiswas prespecifiedA statement about the weakness of the technique as well as anestimate of the effect of using expert opinion on the results of astudy



the panel can provide accurate estimates of transi-tion probabilities, resource utilisation, costs or what-ever is under question.This type of validation study need not be costly.

There are many phase III ‘piggyback’ trials underdevelopment and it would be relatively inexpen-sive to form a panel to provide estimates of thesame parameters included in the placebo arm. Theexpert estimates can then be compared to the ‘goldstandard’ clinical trial data to determine the levelof agreement.Another issue that arises when discussing the

validity of an estimate is its reliability. Reliabilitytesting is of limited use in a Delphi panel as theprocess allows for, and encourages, the revision ofinitial and subsequent estimates. Reliability testingof the entire panel after consensus is reached wouldalso be of limited use. While it might be worth-while to examine this in the short term, panel esti-mates would probably change over themedium andlong term as panellists encounter new informationon the therapeutic area under question and on tech-nological advancements.

4. Conclusion

Further research is needed on the use of expertopinion in pharmacoeconomic studies. In particu-lar, validation exercises as described in section 3.5would increase our knowledge of the precision ofestimates for various sets of information: resourceutilisation, treatment patterns, direct costs, indirectcosts and probabilities. These prospective valida-tion studies would go a long way in determiningwhich categories of data may be collected and whichelicitation technique yields the most accurate infor-mation.Although it is premature to make a complete rec-

ommendation as to the usefulness of expert opinionin pharmacoeconomic evaluations, existing guide-lines and ones under development need to incorpo-rate more information on the potential weaknessesand strengths of the techniques. The absence of anyguidance may be interpreted to mean that since theuse of expert opinion is not proscribed it may beemployed with little concern for itsmethodological

weaknesses. Often these guidelines are used bypeople with little familiarity with economic evalu-ations, and some consideration should be given toestablishing a basic framework for the methods ofdata collection as well as the content of the evalu-ations.Expert opinion is probably here to stay in phar-

macoeconomic research, if only because of neces-sity. The careful use of panel data, correct use ofthe terms and future validation exercises may pro-vide some comfort that its use is not misguided andthat it can play an important role in health eco-nomic research.

References1. Evans C. The use of consensus methods and expert panels in

pharmacoeconomic studies: practical applications and meth-odological shortcomings. Pharmacoeconomics 1997; 12 (2 Pt1): 121-9

2. Commonwealth Department of Human Services and Health.Guidelines for the pharmaceutical industry on preparation ofsubmissions to the pharmaceutical benefits advisory commit-tee. Canberra: Australian Government Publishing Service, 1995

3. Canadian Coordinating Office for Health Technology Assess-ment. Guidelines for the economic evaluation of pharmaceu-ticals: Canada. 2nd ed. Ottawa (ON): Canadian CoordinatingOffice for Health Technology Assessment (CCOHTA); 1997

4. Jones J, Hunter H. Consensus methods for medical and healthservices research. BMJ 1995; 311: 376-80

5. Stewart D, Shamdasani P. Focus groups: theory and practice.Applied social researchmethods series, vol. 20. London: SagePublications, 1990

6. Moore C. Group techniques for idea building (2nd edition):applied social research methods series. London: Sage Publi-cations, 1994

7. Canadian Coordinating Office for Health Technology Assess-ment. A guidance document for the costing process. Version1.0. Ottawa (ON): Canadian Coordinating Office for HealthTechnology Assessment (CCOHTA); 1996

8. Baladi JF, Menon D, Otten N. Use of economic evaluation guide-lines: 2 years experience in Canada. Health Economics 1998;7: 221-7

9. National Health Insurance Council. Draft report on the phar-macoeconomic research guideline. TheNetherlands:NationalHealth Insurance Council , 1999

10. Gold MR, Seigel JE, Russell LB, et al. Cost-effectiveness inhealth and medicine. NewYork (NY): Oxford University Press,1966

11. Barr JT, Schumacher GE. Using decision analysis to conductpharmacoeconomic studies. In: Spilker B, editor. Quality oflife and pharmacoeconomics in clinical trials. 2nd ed. Phila-delphia (PA): Lippincott-Raven, 1996

12. Nuijten MJ, Pronk M, Borerens MJ, et al. Reporting format foreconomic evaluation part II: focus onmodelling studies. Phar-macoeconomics 1998; 14 (3): 259-68



13. Halpern M, Luce B, Brown R, et al. Health economic modelingpractices: a suggested framework. Value Health 1998; 1 (2):131-47

14. DalkeyN, HelmerO.An experimental application of theDelphimethod to the use of experts. Manage Sci 1963; 9 (3): 458-67

15. Guest J, Hart W, Cookson R, et al. Cost analysis of palliativecare for terminally ill cancer patients in the UK after switch-ing from weak to strong opioids. Pharmacoeconomics 1998;14 (3): 285-97

16. Linstone H, Turoff M, editors. The Delphi method: techniquesand applications. Reading (MA): Addison-Wesley PublishingCompany, 1975

17. Schiebe M, SkutschM, Schofer J. Experiments in Delphi meth-odology. In: Linstone H, TuroffM, editors. The Delphi method:techniques and applications. Reading (MA): Addison-WesleyPublishing Company, 1975

18. Jones C. Delphi evaluation of agreement between organiza-tions. In: Linstone H, Turoff M, editors. The Delphi method:techniques and applications. Reading (MA): Addison-WesleyPublishing Company, 1975

19. Antczak-Bouckoms AA,WeinsteinM. Cost-effectiveness anal-ysis of periodontal disease control. J Dent Res 1987; 66 (11):1630-5

20. De Lissovoy G, Rentz AM, Dukes EM, et al. The cost-effec-tiveness of a new chlorhexidine delivery system in the treat-ment of adult periodontitis. J Am Dent Assoc 1999; 130 (6):855-62

21. Sackman H. Delphi critique: expert opinion, forecasting andgroup process. Lexington (MA): Lexington Books, 1975

22. Tversky A, Kahneman D. Judgement under uncertainty: heuris-tics and biases. Science 1974 Sep; 185: 1124-31

23. Nuijten M, Brorens M, Hekster Y, et al. Reporting format foreconomic evaluation: part I. Application to theDutch healthcaresystem. Pharmacoeconomics 1998; 14 (2): 159-63

24. Wilson DA, Gammans RE, Evans CJ. Cost of ischemic strokeby functional status: a model comparing citicoline to standardcare. Value Health 1999; 2 (3): 152

Correspondence and offprints: Dr Christopher Evans, MAPIValues USA LLC, 15 Court Square, Suite 620, Boston, MA02108, USA.E-mail: [email protected]



Expert Judgement in Pharmacoeconomic Studies

Documents

Transcript of Expert Judgement in Pharmacoeconomic Studies