Eliciting Public Preferences for Healthcare

Eliciting public preferences forhealthcare: a systematic review of techniques

M RyanDA ScottC ReevesA BateER van TeijlingenEM RussellM NapperCM Robb

HTAHealth Technology Assessment NHS R&D HTA Programme

Health Technology Assessment 2001; Vol. 5: No. 5

Methodology

Copyright notice

© Queen's Printer and Controller of HMSO 2001 HTA reports may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising Violations should be reported to [email protected] Applications for commercial reproduction should be addressed to HMSO, The Copyright Unit, St Clements House, 2–16 Colegate, Norwich NR3 1BQ

© Queen's Printer and Controller of HMSO

How to obtain copies of this and other HTA reportsCopies of this report can be obtained by writing to:

The National Coordinating Centre for Health Technology Assessment,Mailpoint 728, Boldrewood,University of Southampton,Southampton, SO16 7PX, UK.

Or by faxing us at: +44 (0) 23 8059 5639

Or by emailing us at: [email protected]

Or by ordering from our website: http://www.ncchta.org

NHSnet: http://www.hta.nhsweb.nhs.uk

The website also provides information about the HTA Programme and lists the membership ofthe various committees.

Eliciting public preferences forhealthcare: a systematic review of techniques

M Ryan1 *

DA Scott1

C Reeves1,2

A Bate1

ER van Teijlingen2

EM Russell2

M Napper1

CM Robb1

1 Health Economics Research Unit, University of Aberdeen, UK2 Department of Public Health, University of Aberdeen, UK

*Corresponding author

Competing interests: none declared

Published March 2001

This report should be referenced as follows:

Ryan M, Scott DA, Reeves C, Bate A, van Teijlingen ER, Russell EM, et al. Eliciting publicpreferences for healthcare: a systematic review of techniques. Health Technol Assess2001;5(5).

Health Technology Assessment is indexed in Index Medicus/MEDLINE and Excerpta Medica/EMBASE. Copies of the Executive Summaries are available from the NCCHTA website(see opposite).

NHS R&D HTA Programme

The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health

technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.

Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnosticsand imaging, population screening, methodology) helped to set the research priorities for the HTAProgramme. However, during the past few years there have been a number of changes in and aroundNHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) andthe creation of three new research programmes: Service Delivery and Organisation (SDO); New andEmerging Applications of Technology (NEAT); and the Methodology Programme.

Although the National Coordinating Centre for Health Technology Assessment (NCCHTA)commissions research on behalf of the Methodology Programme, it is the Methodology Group that now considers and advises the Methodology Programme Director on the best research projects to pursue.

The research reported in this monograph was funded as project number 96/49/04.

The views expressed in this publication are those of the authors and not necessarily those of theMethodology Programme, HTA Programme or the Department of Health. The editors wish toemphasise that funding and publication of this research by the NHS should not be taken as implicitsupport for any recommendations made by the authors.

Criteria for inclusion in the HTA monograph seriesReports are published in the HTA monograph series if (1) they have resulted from workcommissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality asassessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search,appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permitthe replication of the review by others.

Methodology Programme Director: Professor Richard LilfordHTA Programme Director: Professor Kent WoodsSeries Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay

and Dr Ruairidh MilneMonograph Editorial Manager: Melanie Corris

The editors and publisher have tried to ensure the accuracy of this report but do not accept liabilityfor damages or losses arising from material published in this report. They would like to thank thereferees for their constructive comments on the draft document.

ISSN 1366-5278

© Queen’s Printer and Controller of HMSO 2001

This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journalsprovided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising.

Applications for commercial reproduction should be addressed to HMSO,The Copyright Unit, St Clements House, 2–16 Colegate,Norwich, NR3 1BQ.

Published by Core Research, Alton, on behalf of the NCCHTA.Printed on acid-free paper in the UK by The Basingstoke Press, Basingstoke. M


List of abbreviations .................................... i

Executive summary ..................................... iii

1 The purpose and plan of this review ........ 1

2 Systematic review of techniques .............. 3Identifying the techniques ............................ 3Establishing the criteria by which to judge the methodological status of techniquesidentified ........................................................ 4Assessing the techniques ............................... 4

3 Techniques identified in the systematic review ......................................... 5Issues raised in the identification of techniques .................................................. 5Quantitative techniques for eliciting public views .................................................... 5Qualitative techniques for eliciting public views .................................................... 13Discussion and conclusions ........................... 16

4 Criteria for assessing the methodologicalstatus of techniques ..................................... 17Criteria for assessing quantitative techniques ...................................................... 17Criteria for assessing qualitative techniques ...................................................... 18Discussion and conclusion ............................ 23

5 A review of quantitative techniques foreliciting public views .................................... 25Ranking techniques ....................................... 25Rating techniques .......................................... 27Choice-based techniques ............................... 31Discussion and conclusion ............................ 38

6 A review of qualitative techniques for eliciting public views ................................... 41Individual approaches ................................... 41Group approaches ......................................... 46Discussion and conclusion ............................ 55

7 The importance of public views in priority setting: the perspective of the policy-maker........................................... 57Methods .......................................................... 57Results ............................................................. 59Discussion and conclusions ........................... 63

8 Discussion and conclusions ........................ 65Results from the systematic review................ 65Results from primary research ...................... 66

9 Recommendations on the use of techniques and future research ................. 71Guidance on the use of techniques .............. 71Future research .............................................. 71

Acknowledgements ..................................... 73

References ..................................................... 75

Appendix 1 Electronic database searchstrategies for identifying methods for eliciting public preferences .......................... 97

Appendix 2 Additional data for chapter 5 ......................................................... 99

Appendix 3 Additional data for chapter 6 ........................................................ 145

Appendix 4 Establishing the relative weights of methodological criteria whenevaluating research techniques ..................... 153

Appendix 5 Summary of criteria used in priority setting and priority-settingframeworks ..................................................... 157

Appendix 6 Questionnaire .......................... 161

Appendix 7 Telephone interview schedule .......................................................... 175

Health Technology Assessment reportspublished to date ......................................... 179

Methodology Group .................................... 185

HTA Commissioning Board ...................... 186

Contents


i

List of abbreviations

AHP analytic hierarchy process

AIC Akaike’s information criterion*

CA conjoint analysis

CE closed-ended

CHC community health council

EUT expected utility theory

HCM health council member

HE health economist

HSR health service researcher

IVF in vitro fertilisation

MA meta-analysis

MoV measure of value

MS medical sociologist

ns not significant

OE open-ended

PC payment card

PEM priority evaluator method

PGI patient-generated index

PHC public health consultant

PTO person trade-off

QALY quality-adjusted life-year

QDP qualitative discriminant process

RPS random paired scenarios

SDT semantic differential technique

SEIQoL schedule for the evaluation of individual quality of life

SEIQoL–DW schedule for the evaluation of individual quality of life – direct weighting

SERVQUAL service quality

SF-36 short-form 36

SG standard gamble

TTO time trade-off

VAS visual analogue scale

WTA willingness to accept

WTP willingness to pay

* Used only in tables


iii

BackgroundLimited resources coupled with unlimited demandfor healthcare mean that decisions have to bemade regarding the allocation of scarce resourcesacross competing interventions. Policy documentshave advocated the importance of public views asone such criterion. In principle, the elicitation ofpublic values represents a big step forward. How-ever, for the exercise to be worthwhile, usefulinformation must be obtained that is scientificallydefensible, whilst decision-makers must be ableand willing to use it.

Aims and objectives

The aim was to identify techniques that could bereasonably used to elicit public views on the pro-vision of healthcare. Hence, the objectives were:

• to identify research methods with the potentialto take account of public views on the deliveryof healthcare

• to identify criteria for assessing these methods• to assess the methods identified according to the

predefined criteria• to assess the importance of public views vis-à-vis

other criteria for setting priorities, as judged bya sample of decision-makers

• to make recommendations regarding the use of methods and future research.

Methods

A systematic literature review was carried out toidentify methods for eliciting public views. Criteriacurrently used to evaluate such methods wereidentified. The methods identified were thenevaluated according to predefined criteria.

A questionnaire-based survey assessed the relativeimportance of public views vis-à-vis five othercriteria for setting priorities: potential health gain; evidence of clinical effectiveness; budgetaryimpact; equity of access and health statusinequalities; and quality of service. Two techniqueswere used: choice-based conjoint analysis andallocation of points technique. The questionnaire

was sent to 143 participants. A subsample wasfollowed up with a telephone interview.

Results

The methods identified were classified asquantitative or qualitative.

Quantitative techniquesQuantitative techniques, classified as ranking,rating or choice-based approaches, were evaluated according to eight criteria: validity;reproducibility; internal consistency; accept-ability to respondents; cost (financial andadministrative); theoretical basis; whether thetechnique offered a constrained choice; andwhether the technique provided a strength of preference measure.

Regarding ranking exercises, simple rankingexercises have proved popular, but their results areof limited use. The qualitative discriminant processhas not been used to date in healthcare, but maybe useful. Conjoint analysis ranking exercises didwell against the above criteria.

A number of rating scales were identified. The visual analogue scale has proved popularwithin the quality-adjusted life-year paradigm, but lacks constrained choice and may not measure strength of preference. However, conjoint analysis rating scales performed well. Methods identified for eliciting attitudes includeLikert scales, the semantic differential technique,and the Guttman scale. These methods provideuseful information, but do not consider strength of preference or the importance of different com-ponents within a total score. Satisfaction surveyshave been frequently used to elicit public opinion.Researchers should ensure that they constructsensitive techniques, despite their limited use, or else use generic techniques where validity has already been established. Service quality(SERVQUAL) appears to be a potentially useful technique and its application should be researched.

Three choice-based techniques with a limitedapplication in healthcare are measure of value,

Executive summary

Executive summary

iv

the analytical hierarchical process and theallocation of points technique, while those morewidely used, and which did well against the pre-defined criteria, include standard gamble, timetrade-off, discrete choice conjoint analysis andwillingness to pay. Little methodological work is currently available on the person trade-off.

Qualitative techniquesQualitative techniques were classified as eitherindividual or group-based approaches. Individualapproaches included one-to-one interviews, dyadic interviews, case study analyses, the Delphitechnique and complaints procedures. Group-based methods included focus groups, conceptmapping, citizens’ juries, consensus panels, public meetings and nominal group techniques.

Six assessment criteria were identified: validity;reliability; generalisability; objectivity; acceptabilityto respondents; and cost.

Whilst all the methods have distinct strengths and weaknesses, there is a lot of ambiguity in theliterature. Whether to use individual or groupmethods depends on the specific topic beingdiscussed and the people being asked, but for both it is crucial that the interviewer/moderatorremains as objective as possible. The most popularand widely used such methods were one-to-oneinterviews and focus groups. Both methods havepotential problems with validity and reliability, and the researcher must minimise these problemsat all stages of data collection, analysis and dis-semination. The Delphi technique was widely used, but participants were only occasionallypatients. It is proposed that the Delphi techniquecould be more widely used to gain patients’opinions. Citizens’ juries were found to be veryuseful, especially with complex subject matter; the final decisions and opinions of participants are particularly valid and reliable because of theopportunity for deliberation. However, there areproblems with generalisability as only very smallnumbers of people can be involved; they are alsovery time-consuming and therefore costly. Con-sensus panels are similar to citizens’ juries but do not allow sufficient time for decision-making.They are less costly so cannot be dismissed. Publicmeetings, which are frequently used and are aquick and inexpensive way of gaining publicopinion, are unrepresentative. Finally, complaintsprocedures only consider negative viewpoints, and therefore have limited value. Further work isneeded to establish the appropriateness of casestudy analyses, concept mapping and nominalgroup techniques.

The importance of public views inpriority settingBoth the choice-based conjoint analysis techniqueand the allocation of points method found thepublic’s views to be important in the priority-setting exercise, although the relative rankingsdiffered across the two techniques. In the follow-up telephone interviews, whilst the majority ofrespondents stated that the community had a role to play in decision-making, and that this rolewas (very) important in the context of prioritysetting, they ranked it as the least important of the six criteria.

Conclusions

Recommended techniquesThere is no single, best method to gain publicopinion. The method must be carefully chosen and rigorously carried out in order to accommo-date the question being asked. Conjoint-basedmethods (including ranking, rating and choice-based), willingness to pay, standard gamble andtime trade-off of the quantitative techniques andone-to-one interviews, focus groups, Delphitechnique and citizens’ juries of the qualitativemethods are recommended. Likert scales, thesemantic differential technique and Guttmanscales are useful quantitative techniques foreliciting attitudes and knowledge.

Recommendations for future researchResearching techniques:• the techniques recommended above should

continue to be researched• research to investigate analytical hierarchical

process, measure of value, allocation of points,the qualitative discriminant process, SERVQUALand person trade-off as quantitative methodswith telephone, email and dyadic interviewtechniques, consensus panels, case studyanalyses, concept mapping and nominal group techniques as qualitative methods

• when addressing the above points, a priorityarea of research is to address the extent towhich preferences for healthcare exist, as well as the cognitive strategies and decision-making heuristics respondents adopt whencompleting quantitative surveys. This shouldinvolve extensive qualitative work to inform the design and interpretation of quantitative studies.

General issues raised in the review:• do the public want to be involved in healthcare

decision-making?


v

• potential problems encountered with apreference for the status quo

• ethical issues in involving the public

• developments of frameworks to ensure public preferences are incorporated into priority setting.


1

Limited resources coupled with unlimiteddemand for healthcare mean that decisions

have to be made regarding the allocation of scarceresources across competing interventions. Numer-ous criteria may be used to help this decision-making process. A combination of factors, includ-ing the rise of certain social movements (public,patients and feminist) and changes in the organis-ation of the NHS, has led to a growing awarenessthat the views of the public need to be taken intoaccount. Policy documents agree.1–4 For example,in its publication Local Voices,1 the NHS Manage-ment Executive has advocated the need to takeinto account the views of local people when setting healthcare priorities.

In principle, attempts to involve public views in priority setting represents a big step forward.However, for the exercise to be worthwhile usefulinformation must be obtained that is scientificallydefensible and decision-makers must be willing to use it.

The aim of this project was to identify techniquesthat could be reasonably used to elicit public viewsin the provision of healthcare at both the microand macro level.

Following this aim, the objectives were:

• to identify research techniques with thepotential to take account of public views in the priority-setting exercise in healthcare

• to identify criteria for assessing these techniques• to assess the techniques identified according to

the predefined criteria• to assess the importance of public views vis-à-vis

other criteria for setting priorities, as judged bya sample of decision-makers

• to make recommendations regarding the use oftechniques and future research.

The plan of this report is as follows. Chapter 2 sets out the methods used to address the first three objectives listed above. Following this,chapter 3 presents descriptions of the techniquesidentified in the review and chapter 4 identifies the criteria by which these techniques wereassessed. Chapter 5 then presents the review of the quantitative techniques and chapter 6 thereview of the qualitative techniques. Chapter 7presents the methods and results of the primaryresearch looking at the importance of publicpreferences in relation to other criteria for settingpriorities (fourth objective above). Chapter 8makes some overall conclusions and chapter 9presents recommendations regarding the use of techniques and future research (the lastobjective above).

Chapter 1

The purpose and plan of this review


3

Systematic reviews are “a scientific tool whichcan be used to summarise, appraise, and com-

municate the results and implications of otherwiseunmanageable quantities of research”.5 The reviewpresented here is concerned with methodologicalresearch, or, more specifically, with defining bestpractice in the area of eliciting public preferencesin the provision of healthcare (as opposed todefining best medical practice). Methodologicalsystematic reviews have become increasinglyimportant in recent years,6 leading to the questionof what ‘systematic’ means for such reviews. Whatis clear is that methodological reviews have tendedto take different approaches, both to identifyingrelevant material and to drawing conclusions fromit. This has resulted from the broad range of issuesaddressed, with the nature of the review being very dependent on the question being addressed.Below we outline the methods used in our syste-matic methodological review to identify methodsfor eliciting public preferences in the delivery ofhealthcare. This was divided into three main parts:identifying the techniques; establishing the criteriaby which to judge the methodological status ofmethods identified; and assessing the methods.

Identifying the techniques

Given the aim was to identify methods that hadbeen used both within and outside health, a rangeof databases was searched. The core bibliographicdatabases used across all searches were MEDLINE(1966–), EMBASE (1980–), HealthSTAR (1975–)and the Social Science Citation Index (1981–). In addition, PsycLIT and the economics databaseEconLIT (1969–) was used extensively throughout.Other databases used were the Health Manage-ment Information Consortium Database (HMIC),CINAHL, Sociological Abstracts, and the Instituteof Management International Databases (IMID).Search strategies were formulated using appro-priate combinations of controlled vocabulary(where available) and free text terms. The search strategies are given in appendix 1.

Particular attention was paid to searching the greyliterature, for example government publicationsand research reports through use of the HealthManagement Information Consortium Database,SIGLE (System for Grey Literature in Europe),IDEAS (Internet Documents in Economics Access Service), the Scottish Health ServiceManagement Library’s in-house database and the University of Aberdeen Health EconomicsResearch Unit’s own specialised library resource.The Health Service Management Centre’sdatabase, ‘International Approaches to PrioritySetting in Health Care’ and the University of York’s database of examples of consumerinvolvement in research were also consulted.

The Internet was searched using specific relevantwebsites and through a search engine (chieflyAltaVista®), where considered appropriate. TheSocial Science Information Gateway (SOSIG) was used to identify quality-assessed websitesrelevant to public opinion research. The InternetDocuments in Economics Access Service was used to identify any relevant economics-basedresearch reports.

Bibliographic searching was supplemented bycitation searching of key articles, reviewing ofreferences from key publications, and byconsulting key texts.7–12

Following this initial search, the abstracts werescanned by the researchers (DAS for quantitativemethods and CR for qualitative methods) andthose that identified techniques that could be used for eliciting public preferences were obtainedin hard copy. All other articles identified by thescope of the search strategy, for example thosediscussing the methodology of priority setting or patient involvement in decision-making, wereexcluded from the systematic review*. Papers notwritten in English were excluded. This resulted ina number of potentially useful techniques beingidentified which were examined in more detail.The authors then sought advice from a number

Chapter 2

Systematic review of techniques

* Where such articles highlighted criteria and frameworks that are currently used for priority setting or general issuesthat are raised when eliciting public preferences, these were obtained in hard copy. The issues raised are covered inchapters 7 and 8 of this report.

Systematic review of techniques

4

of experts about whether any techniques couldhave been omitted from their list.

A secondary search was then conducted whichfocused on each individual technique. Thisconsisted chiefly of straightforward searchingacross databases using the name of each techniqueas free text terms (unless available as controlledterms). For individual techniques, any acronyms,synonyms and any commonly used shorthandversion of the name of the technique wereincluded as free text terms. The search engineAltaVista was again used when searching formaterial relevant to a particular technique. Anattempt was made to identify studies that had both applied the technique in healthcare, and had, either explicitly or implicitly, discussedmethodological issues. Where the technique had been used in health, only those articles were considered but where the technique had not been used in health, the authors digressedfurther afield.

Following this secondary search, the abstracts ofany additional papers were reviewed and if deemedrelevant, a hard copy was obtained. No handsearching was undertaken within this secondarysearch. However, the reference lists of identifiedarticles were searched for relevant studies and, in some cases, individual specialists in particulartechniques were consulted for their reference lists. For example, one of the authors (MR) hadconsiderable experience in the use of conjointanalysis (CA) and willingness to pay (WTP) andhad colleagues who had also researched thoseareas. They were able to identify additionalliterature such as conference papers and works in progress. In addition, further searches wereconducted on key individuals who were especiallyknown for or who had contributed significantly to the adoption of a particular technique. Forexample, James Dolan had done a considerableamount of work on the analytic hierarchicalprocess (AHP) and Eric Nord had also frontedmuch of the work on person trade-off (PTO). For techniques where little literature was found, for example Hoinville’s priority evaluator method (PEM), articles citing the key papers were sourced. Any additional articles brought to light were obtained in hard copy.

All retrieved references were incorporated into a dedicated database using the bibliographicsoftware package Reference Manager©.

Establishing the criteria by whichto judge the methodologicalstatus of techniques identifiedAlthough initially we did not discriminate betweenquantitative and qualitative methods for elicitingthe views of the public, the literature search con-firmed at an early point that methodologicalevaluation is at very different stages of formalis-ation for the two approaches†. Moreover, there is an ongoing debate about whether it is possible to apply the same criteria to both.13 We thereforedecided to handle them separately. We attemptedto summarise the current debate on assessingquality in quantitative and qualitative research. To do this, articles identified in the systematicliterature review described above were used, as well as additional literature the authors werefamiliar with. Following this, we defined thecriteria by which the methods identified in thereview were to be assessed.

Assessing the techniques

The quantitative and qualitative techniques identi-fied in the searches were then reviewed accordingto predefined criteria. This formed the completionof the systematic review of the techniques. Thisproject used only one researcher to review thetechniques using the predefined criteria (althoughseparate researchers were used for the quantitativeand qualitative reviews). Whilst it is recognised that using more than one researcher to evaluatetechniques would reduce researcher bias, time and resource constraints meant this was notpossible. It may also be argued that, given thenature of the review (i.e. there were well-definedcriteria with which to assess the techniques),interobserver reliability was not really a concern.However, this argument may apply more for thereview of quantitative techniques than qualitativetechniques. The synthesis of the results is quali-tative, highlighting the strengths and weaknesses of the different approaches.

† Quantitative instruments allow estimation of ‘numbers’. In contrast, qualitative instruments “begin with an intentionto explore a particular area, collect ‘data’ and generate ideas and hypotheses from these data largely throughinductive reasoning”.649 Qualitative research seeks to find answers to the ‘why’ questions as opposed to the ‘how many’questions. Such methods help to find out what the ‘meaning’ of a certain phenomenon, event or relationship, forexample, is for people and the ‘context’ in which these phenomena are experienced. For example, quantitativeresearch may reveal that respondents hold certain attitudes, and qualitative research why they hold these attitudes.


5

Issues raised in the identification of techniquesGiven the nature of the topic to be reviewed, it was apparent from the outset that a ‘Cochrane-type’ approach to identification of the relevantliterature was not appropriate.14 However, asystematic and comprehensive strategy was used.The initial strategy was designed to trawl theliterature covering a number of differentdisciplines, both within and outwith the health-care field. This was reflected in the choice ofbibliographic databases. Because the literature was an unknown quantity at the initial stage, the approach to formulating the search strategywas progressive rather than algorithmic. Thisapproach is consistent with other methodo-logical reviews.6,14

The search strategy was therefore developed (using a mix of controlled vocabulary and free text terms) to give high recall and therefore, bydefinition, low precision.5 Consistent with othermethodological reviews,6,14,15 this approach resultedin a large number of articles for some techniquesand a limited amount for others. We did notpursue all references, but made an attempt to stop the search when it appeared (from theabstracts) that the new literature emerging was of no additional benefit. Edwards and colleagueshave referred to this truncation process as“theoretical saturation”. They note that it is“important to cast the net wide by searching many types of literature to make sure that aparticular line of argument was not missed, rather than to pursue every instance of theargument”. Without such a strategy in this study it would not have been feasible to complete the project within the deadline.

It also became clear from the search strategy that whilst searching multiple databases lead to the identification of a wide range of techniques, it did so at the cost of duplication of references.

However, this was deemed necessary by theresearch team if techniques were to be identifiedfrom a range of disciplines.

When searching for articles dealing withmethodological issues such as validity/reliability,some databases included appropriate controlledthesaurus/vocabulary terms, for example inMEDLINE an appropriate MeSH term would be“Reproducibility of results”. However, sometimes, a controlled vocabulary term was not available.Free text truncated terms were then used. Thislatter search technique, given that it searches for words regardless of context, resulted in a largeamount of irrelevant references. However, thismethodology was again deemed necessary by the research team.

Whilst the potential problems of a methodologicalsystematic review were realised in this project, theproject nevertheless resulted in information on awide range of techniques that have been used, ormay be used, to elicit public preferences in thedelivery of healthcare. These techniques aredescribed below. Most have been used to someextent in healthcare, though the frequency withwhich they have been applied varies widely*.Quantitative techniques are first described,followed by qualitative techniques.

Quantitative techniques foreliciting public viewsThe methods identified have been classified as ranking, rating or choice-based techniques.Appendix 2 presents a summary of studiesidentified (and methodological issues addressed;see chapter 5).

Ranking techniquesSimple ranking exerciseRanking exercises in their simplest form askrespondents to give an ordinal ranking to

Chapter 3

Techniques identified in the systematic review

* Community health councils (CHCs) have been mentioned in the area of measuring public preferences.650 Theirremit is to consult with and represent the public, monitor services, and give advice and information.651 However,CHCs are not a technique or instrument for obtaining public opinion, but a vehicle to aid the identification ofconcerns and attitudes of the public. Therefore, CHCs are not included in this review.


6

options†. Those options that achieve the highestranking are viewed as the most important. Giventheir relative ease to analyse and complete byrespondents, this approach has proved popular as a method of eliciting public preferences inhealthcare.16–27 The technique has been applied ina number of different contexts, including elicitingpreferences for healthcare interventions, lookingat the importance of patient characteristics inchoices concerning who receives care, and valuingdifferent aspects of clinical outcomes. For example,Groves17 asked a random sample of the generalpublic, managers and doctors in the UK to rankten diverse healthcare interventions in order ofpriority for spending. Responses were averaged,and a ranking estimated for each of the teninterventions. Bowling and colleagues16 adopted a slightly different ranking approach. In a studyexamining the attitudes towards priority setting inan inner London health district, the public andhealth professionals were asked to rank 16 healthservices in relation to their views on the needs ofpeople in City and Hackney. The respondents were required to rank four services as ‘essential’,four as ‘most important’, four as ‘important’ andfour as ‘less important’. An overall priority scorewas estimated for each service/area/treatment bycoding the possible responses from 1 (essential) to 4 (less important) and then averaging over these responses.

Qualitative discriminant processThe qualitative discriminant process (QDP) is a scoring and ranking process that has beendeveloped in the business environment.28–30 Thetechnique is based on decision analysis techniquesand is computer based. The distinguishing featureof this technique is that it involves moving fromdefining options in terms of qualitative categories,through to deriving a numeric point estimate, and finally solving a maximisation problem within given constraints. The technique involves three steps:

Step 1: the respondent is guided through threecomputer-based ranking rounds where they areasked to rank scenarios according to predefinedqualitative descriptions. These three ranking

stages are defined as broad, intermediate andnarrow. Round 1 may involve defining scenariosas ‘very high’, ‘high’, ‘average’ and ‘low’. Thisround will provide a preliminary ranking ofalternatives. In round 2, respondents arerequired to further differentiate alternativesaccording to further subcategories, for example‘top’, ‘middle’ and ‘bottom’. The same processhappens in round 3, with respondents nowbeing asked to rank each alternative into further sub-subcategories, i.e. ‘upper’ and‘lower’. At each round, if two alternatives aredefined equally, pairwise comparisons are usedto determine the ranking. Following these threerounds, a ranking of alternatives is provided.

Step 2: this involves mapping the qualitativeresponses onto a numeric interval scale, andproducing a vague real number for all options.

Step 3: linear programming techniques are used to identify the optimal solution within the given constraints.

Bryson and colleagues28 provide an illustrativeexample within the context of identifying suitablecandidates for the position of Dean of the BusinessSchool. The three criteria chosen to distinguishcandidates were academic credential, managementcredentials and fundraising credentials. In thisillustration, a multistage decision-making processwas used, whereby the candidates were consideredonly if they had excellent academic credentials.The example provided relates to this stage.However, it should be noted that any stage couldinvolve multiple criteria. In round 1, decision-makers were presented with a list of ten candidatesand asked to rank them into the broad categoriesof ‘excellent’, ‘very good’, ‘satisfactory’ or ‘poor’.Respondents are encouraged to move the candi-dates around until they are completely satisfied.The visual computer screens help this process. This stage yields a preliminary ranking of altern-atives. Round 2 involves a second ranking exercise,whereby the decision-makers were asked to furtherdistinguish the candidates into ‘top’, ‘middle’ and‘bottom’. Again, respondents are encouraged tomove candidates around. Round 3 involves further

† Variations around such a simple ranking exercise were identified in the literature. Plurality ranking involves therespondent awarding their chosen option one point and the others zero. Options are summed for all respondents toachieve a rank order.652 The Borda measure, derived from social choice theory, involves allocating zero points to theoption ranked lowest, one point to the next lowest and so on until the highest ranked option receives the highestpoints. Scores are then weighted by numbers of respondents to give the Borda score for each alternative. This Bordascore will take on a value between 0 and 1; the highest Borda score being the most desired option.652,653 No studieswere identified which had used this approach in healthcare.


7

differentiation into ‘upper’ and ‘lower’. Fromthese three rounds individual candidates areranked. For example, the top candidate would bethe one defined as ‘excellent’, ‘top’ and ‘upper’. If candidates were defined equally then thedecision-maker would have to make choices.Following the ranking exercise, a score for eachcandidate was assigned using an interval scale from 0 to 100, and defined as excellent (80–100),very good (60–80), satisfactory (30–60) and poor (0–30). Any subdivision of the ranking scale could be used.

The QDP is in the early stages of development, and there have been no applications elicitingpublic views in healthcare.

CA ranking exercisesCA (whether it adopts a ranking, rating or choice-based approach; see below for a descriptionof the latter two) is rooted in Lancaster’s theory of value.31 This theory assumes that goods can bedescribed by their characteristics, and that thesegoods enter an individual’s benefit (utility) func-tion as a combination, whereby the total utilitygained from consuming a good is a function of the individual utilities from the characteristics of that good. The technique was developed inmathematical psychology,32 and has been widelyused in market research,33 transport economics,34

and environmental economics.35,36 The techniqueis now gaining more widespread use in healtheconomics to elicit public preferences for health-care interventions. The technique involves fivemain stages:

Stage 1: identification of attributes,characteristics or criteria that are important in achieving the overall stated objective of the study.Stage 2: assigning levels to these criteria.Stage 3: using experimental designs to reducethe number of scenarios that individuals arepresented with down to a manageable level.Stage 4: eliciting preferences using ranking,rating or choice exercises.Stage 5: analysing the data using regressiontechnique.

From this it is possible to estimate: the marginalbenefit or weight of individual criteria; the rate atwhich individuals trade between these criteria; andthe overall benefit (or utility) scores for differentcombinations of levels of criteria. This latter scoreis achieved by multiplying the criterion weight bythe level, and summing across all criteria for thegiven service.

A number of studies were identified that had usedranking-based CA exercises in healthcare.21,37–45

Using the ranking approach, individuals are pre-sented with a number of scenarios involving adifferent combination of criteria and levels, andasked to rank these. For example, Singh andcolleagues46 used the ranking CA technique toexamine patient decision-making on growthhormone therapy. Six criteria were identified asimportant: risk of long-term side-effects (1:10,000or 1:1,000,000); certainty of effect (50% or 100%of cases); amount of effects (1–2 inches or 4–5inches in adult height); out-of-pocket cost ($100,$2000, or $10,000/year); route of treatment (daily injections or nasal spray)and child’s atti-tude (likes or does not like therapy). Respondentswere asked to provide a ranking of a number ofscenarios that involved different combinations ofthe above criteria and levels. Further examples ofthe application of the ranking CA approach havebeen to determine physicians’ decision-makingprocess in the context of anti-infective drugs,47

how to increase a hospital’s patient population,38

choosing alternative health plans,21,39 alternativecontraceptive methods,40 evaluation of residents’clinical competence,41 ambulatory caremanagement,43 and preferences for growthaugmentation therapy.46

Rating techniquesRating scales involve presenting individuals withcriteria, scenarios or statements and asking them to respond with respect to their opinions, attitudesor knowledge on either a numerical or semanticscale. Numeric scales usually provide anchordescriptions such as ‘best outcome’ or ‘worst out-come’. A large number of rating scale approacheswere identified in the literature, with such scalesbeing used to address a number of different issues,including estimating quality weights for healthoutcomes, understanding patient preferences fordifferent aspects of treatment and assessing thepublic’s attitudes and knowledge concerningdifferent issues. Economists have tended to userating scales to estimate utilities, whereas othersocial scientists have been more concerned withpublic attitudes. Amongst the latter a commonapplication is satisfaction-type surveys. In thissection we consider the range of such scales that have been used in healthcare to elicit public preferences.

Rating scales within the quality-adjusted life-year literatureRating scales, usually referred to as visual analoguescales (VAS), have been widely used by economiststo estimate utility weights within the quality-


8

adjusted life-year (QALY) literature.48,49 QALYswere developed to take account of the fact that an individual may be concerned with the quality of their life as well as the quantity of life. Toestimate QALYs, expected life years gained fromgiven healthcare interventions are estimated andcombined with information on the quality of theselife years (via the estimation of utilities). Forexample, if a healthcare intervention results in ahealth state with a utility score of 0.85, and theindividual would be in this health state for theremainder of life, say ten years, then the numberof (undiscounted) QALYs would be 8.5. TheQALYs gained from one healthcare interventionmay be compared with QALYs obtained fromalternative healthcare interventions, as well as from doing nothing. The QALY framework wasdeveloped to aid decisions concerning whathealthcare interventions to provide‡.50 Asking the public to estimate quality weights (as opposed to, for example, clinicians) will therefore incorporate public preferences into the decision-making process.

When using the VAS to estimate quality weightswithin the QALY framework, respondents arepresented with a line with anchors at best imagin-able health-state (with a score of 100) and worstimaginable health-state (with a score of 0). Suchapplications have often provided guidance torespondents by providing numbers at intervals.Respondents are then asked to indicate on thisscale the point corresponding to either their own health-state, or another defined health-state.This point is taken as the quality weight for thathealth-state.

Rating scales within CA studiesRating scales have been used in the CA literatureto elicit patient and community preferences inhealthcare. Here respondents are asked to ratescenarios on a numeric or semantic scale. Theoverall score is the dependent variable, and this is regressed against predefined levels ofcriteria.51–65 Within healthcare this technique has been used to address a number of issues,including developing a handicap outcomemeasure,58 establishing preferences for aspects of health services (including general practice,60

scoliosis surgery,51 antihistamine drugs,59

dental health services52), obstetricians’ referralpatterns,66 speciality selection by medicalstudents,54 design of an obstetrics unit,56 factorsinfluencing physicians’ decision to operate,57

physicians’ weighting of clinical information64

and hospital selection decisions.65

Schedule for the evaluation of individual quality of lifeA variation of the CA rating technique, whichshares the same theoretical foundations, is theschedule for the evaluation of individual quality oflife (SEIQoL).67 Using this technique respondentsare first asked to identify the five most importantareas of their life in terms of quality of life. Theindividual then rates each of these areas on a scalewhere the upper extreme is defined as ‘as good asit could possibly be’ (with a score of 100) and thelower extreme as ‘as bad as it could possibly be’(with a score of 0). Computer-generated hypo-thetical cases are then presented to individualswhich represent different combinations of theareas of life specified as important. Respondentsare asked to rate these hypothetical cases on a scale with the same upper and lower extremes asdefined above. Regression techniques are used toestimate the weights for the areas of life, and theseweights are multiplied by the individual’s self-ratingand summed over the five areas to give a quality oflife score. SEIQoL has been used to assess qualityof life in hip replacement patients,68 dementiapatients,67 the elderly69 and sufferers of irritablebowel syndrome.70

Likert scaleWhilst economists have used rating scales toestimate quality weights or benefit scores, othersocial and behavioural scientists have tended tofavour scales that are concerned with respondent’sattitudes. A common technique used here is theLikert scale. This contains a series of opinionstatements on a given issue.9 Respondents’attitudes are elicited by presenting them with aseries of statements and asking them their level of agreement on an agree–disagree continuousscale.71–74 This is often an ‘odd’ number scale, with a neutral/undecided point in the middle.Likert scales have been used in healthcare toaddress a variety of issues including NHSmanagers’ attitudes to capital charging,75

‡ Four techniques have been used in the literature to estimate quality weights: rating scales, magnitude estimation,standard gamble (SG) and time trade-off (TTO). SG and TTO are considered in the next section under choice-basedtechniques. Magnitude estimation is not considered in this report because it has been used infrequently; no morestudies were identified beyond Brazier and colleagues’ review,253 and Brazier and colleagues did not recommend its use.


9

physicians’ or nurses’ views on the quality ofhealthcare,76,77 dentists’ and patients’ judgementsof ideal patients and dentists, respectively,74,78,79

a patient’s perceived quality of life,80 views onpriority setting and public involvement73,81,82

and medical students’ evaluations of patients.83

Likert scales have also been extensively used in satisfaction studies (see below).

Semantic differential techniqueUsing the semantic differential technique (SDT),attitudes are elicited according to a number ofopposite or polar adjectives at each end of ascale.84–87 Responses are subsequently scored togive an overall ‘attitude score’. The SDT, as well asbeing used in satisfaction scales (see below), hasbeen used to assess attitudes regarding a numberof factors in healthcare. These include attitudestowards: the menopause,88,89 diabetes,90 cervicalscreening,84 health-related behaviour,86,87 cancerdetection methods,91 HIV/AIDS92 and schizo-phrenia.93 For example, Swain and McNamara92

carried out a study concerned with Irish pupils’attitudes to HIV/AIDS. The authors utilised thefollowing bipolar adjectival eight-point scales:bad–good, false–true, painful–pleasurable,ugly–beautiful, unfair–fair, unimportant–important, unsafe–safe, worthless–valuable.

Guttman scalesGuttman scales have been applied in the fields of sociology, politics, psychology and consumerresearch to assess attitudes and knowledge.71,94–97

They require respondents to respond to state-ments in terms of ‘agree’ or ‘disagree’. Responsesare coded as ‘yes’ for agree and ‘no’ for disagree.Scores of individual ‘yes’ responses are then estim-ated. From this it is possible to see which questionsrespondents as a group agree with.96 Only fivepapers using the Guttman scale in healthcare were identified. These assessed attitudes towards:alcohol,98 meat avoidance,99 autopsy, organdonation and dissection,100 long-stay care for the elderly101 and dental health.102

Satisfaction surveysPatient satisfaction surveys are a popular methodfor eliciting public opinion in healthcare, both in the UK and elsewhere.73,103–158 These studiesrepresent only a handful from the total numberavailable. Bisset and Chesson159 note that therehave been over 4000 entries on MEDLINE over the past 5 years, as well as many others in the grey literature.

Fitzpatrick144 outlines three reasons for conductingpatient satisfaction studies: (i) an important

outcome measure; (ii) useful in assessing consult-ations and patterns of communication; and (iii)used systematically, feedback enables choicebetween alternatives in organising or providinghealthcare. This review focuses on the latter aspect of patient satisfaction studies. However,satisfaction studies are rarely designed andimplemented purely for this third reason.

Within the UK, a number of patient satisfactionquestionnaires have been developed with the aimof evaluating NHS services nationally. These havebeen concerned with different aspects of care,including hospital care,160 general practice con-sultations,146,147,161 out of hours primary medicalcare,148 screening for breast cancer,162 andmaternity services.163,164 However, most patientsatisfaction surveys are based on questionnairesdeveloped by the researchers themselves.120

Direct or indirect methods may be used to elicitpatient satisfaction. The former method involvesdirectly asking respondents their level of satis-faction with given aspects of care, with possibleresponses being on a rating scale ranging from‘very satisfied’ to ‘very dissatisfied’. Indirectmethods involve inferring satisfaction indirectlyfrom responses to such questions as ‘the doctorprovided me with all the information I wanted’,with possible responses being on a Likert scale or on a Guttman scale (see pages 8 and 9).

SERVQUALService quality (SERVQUAL) was developed in the marketing literature as a generic measure ofservice quality.165,166 This technique is based inmultiple discrepancy theory, and is concerned with the gap between expectations and percep-tions. A Likert scale is used to measure ‘quality’ in terms of this difference. There are fivedimensions to the SERVQUAL, and 22 statementsreferring to these dimensions. Respondents arefirst presented with 22 items concerning expect-ations about quality and asked to agree on a Likert-type scale, with 1 representing stronglydisagree and 7 strongly agree. They are thenpresented with 22 items relating to their per-ceptions concerning the commodity beingevaluated, and again asked to agree on a Likert-type scale, with 1 representing strongly disagreeand 7 strongly agree. Quality is defined in terms of perceptions minus expectations. The resultsfrom such a survey help providers target thoseareas where the gap between expectations andperceptions is greatest. The technique has received much empirical attention in thehealthcare literature.166–182


10

Choice-based techniquesChoice-based techniques force individuals tochoose between alternatives presented to them.Given this, they all incorporate the choicecriterion. Such techniques have taken a variety of forms when eliciting public preferences in healthcare.

Simple choice exercisesChoice exercises, at their simplest level, involvepresenting respondents with scenarios that varywith respect to one characteristic, and asking themto choose between them. Charny and colleagues183

asked respondents to choose between two hypo-thetical individuals differing by only one character-istic (age, marital status, gender, smoker/drinkerstatus and employment). In a subset of this study,Lewis and Charny,184 looking at the importance of age when allocating scarce healthcare resources,presented individuals with three scenarios. Eachscenario forced the respondent to choose betweentwo patients with leukaemia who were alike in all respects other than their age. They were toldthat only one of them could have treatment. A similar approach was adopted by Mooney and colleagues185 in examining preferences forallocating healthcare gains. Here respondents were presented with six choices where equal health gains were to be allocated to differentpopulation groups based upon their age, sex,current health, socio-economic status, timing when benefits would be achieved and the number of individuals who would benefit.

An extension of this technique is random pairedscenarios (RPS). This technique also involvespresenting individuals with paired scenarios andasking them to make choices. However, morecharacteristics are included in the scenarios. Cross-tabulations are used to determine how often each attribute is selected as a ‘winner’. Thehigher the selection rate, the more important thatattribute. One study was identified which appliedthis technique in healthcare. This was concernedwith the importance of different characteristics of patients when deciding on the allocation ofscarce healthcare resources.186 Characteristicsidentified as important were age (child, oldpatient), income (poor, rich patient), severity of disease (mild, severe), prognosis (good, poor),social status (low, high), cost of treatment(inexpensive, expensive) and origin of disease(self-acquired or not). In all, 24 hypotheticalpatients were created, each one containing threedifferent characteristics of patients randomlyselected from the list of six (e.g. child, rich patient, severe disease). Each respondent was

presented with 12 random pairs. For each pair they were asked which one they would choose to be treated if only one could be paid for bysociety. Cross-tabulations were used to calculate the number of times each characteristic wasselected as a winner. A selection rate of over 50% (meaning this characteristic was chosen as a winner in 50% of all the scenarios where it appeared) was defined as meaning that char-acteristic should be prioritised; 50% implied a neutral attitude; and less than 50% a negative prioritisation.

CA choice-based questionsChoice-based questions have been widely usedwithin the CA framework in healthcare. Hereindividuals are presented with choices that involve different combinations of a good orservice, and, for each choice, they state which they would choose or prefer. Possible responsesmay be either discrete (i.e. prefer A or preferB)187–203 or graded (i.e. strongly prefer A, prefer A, indifferent, prefer B, strongly prefer B).204–209

Choice-based CA is gaining widespread use in healthcare and has been applied in a number of areas including: eliciting patient/community preferences in the delivery of healthservices,9,187,191,196,197,210 establishing a consultant’spreferences in priority setting,211 evaluating health-states,212,213 determining optimal treatmentsfor patients,64 evaluating alternatives withinrandomised controlled trials (RCTs)199 andestablishing patient preferences in thedoctor–patient relationship.202,203

Analytic hierarchy processThe AHP has been widely used in the areas ofsocial science, engineering and business to assessthe relative importance (weights) of differentcriteria in the provision of a good or service, andfollowing on from this, to derive scores for givengoods and services.214–224 There are four mainstages involved in conducting an AHP study:220

Step 1 – Structure the problem: the first stage when using AHP is to construct a decision hierarchy for the overall problem being considered. This will usually involvestating the objective (level 1), the criteria that are important in achieving this objective(level 2), and the alternatives that may be used to achieve the objective, which willcomprise different components of the criteria (level 3).

Step 2 – Making choices: the second stepinvolves making relative judgements across


11

adjacent hierarchical levels. This is done with pairwise comparisons. First, the criteriaidentified (level 2) are assessed relative to the overall objective (level 1), and then thealternatives (level 3) are assessed in terms ofeach criterion (level 2). These choices are made in terms of preferences or importance on a nine-point ‘intensity of importance’ scaledeveloped by Saaty.223 This scale has definitionsand explanations attached to the numeric scale, and is held to be on a ratio scale. Forexample, if criteria i and j from level 2 arecompared in terms of their importance inmeeting the objective, and criteria i is consid-ered to be five times more important thanoption j, then j is 1/5 as important as i. Assum-ing n criteria at level 2, then (n(n – 1))/2 arerequired. A pairwise matrix is created, com-paring all criteria with each other. Followingthis, the alternatives (from level 3) are eachconsidered in relation to each of the criteria(from level 2).

Step 3 – Estimating weights: following this,computer software is used to estimate weightsfor both the individual criteria in terms of meeting the overall objective, and thealternatives in terms of meeting the criteria.

Step 4 – Synthesising weights to scorealternatives: following stage 3, a composite score is estimated for each alternative. This ismade up by multiplying the relative weight ofeach criterion (from level 2) by the weight foreach criterion within each alternative, andadding the results.

A number of studies were identified which hadapplied this technique to healthcare, although all were within the context of the Americanhealthcare system§. Whilst the majority of studies have addressed issues relating to clinicaldecision-making and optimal treatment paths for patients,225–237 preferences for alternativehealthcare services238–242 and systems240,243,244

had also been addressed.

Standard gambleSG is a choice-based technique that has beenwidely used by economists within the QALYparadigm. Using SG, a choice is presented whichrequires the respondent to choose between a

certain outcome and a gamble. If the gamble ischosen, it may result in either a better outcome(with a probability p) or a worse outcome than the original (with a probability 1 – p). The utilityweight is gained through adjusting the probabilityof the best outcome until the subject is indifferentbetween the certain intermediate outcome and the gamble.49,245,246 Given the known difficulties of answering a SG-type question, the technique is carried out with visual aids and via face-to-face interviews.

Time trade-offGiven the known difficulties of answering SG ques-tions, the TTO technique was declared to estimateutility weights within the QALY framework.247 Thisapproach involves presenting individuals with achoice between living for a period t in a specifiedbut less than perfect state (outcome B) versushaving a healthier life (outcome A) for a timeperiod h where h < t. Time h is varied until therespondent is in-different between the alternatives.The utility weight given to the less than perfectstate is then h/t.

Person trade-offThe PTO technique may be seen as an extensionto the QALY approach. Whilst the QALY approachvalues the health effects of interventions (obtainedfrom VAS, SG or TTO), the PTO extends this toallow for distributive issues, i.e. who to treat.248–250

The social value of a given healthcare interventionis derived by multiplying the utility gain from agiven healthcare intervention (estimated usingeither VAS, SG or TTO) by both a social weight(SW), which is determined by the severity of theinitial condition, and a potential for health weight(PW). Both social weight and potential for healthweight are estimated using the PTO method. Thisinvolves asking individuals how many outcomes of one kind (x) are equivalent in social value to y outcomes of another kind. These descriptions will vary with respect to the distributive aspects that are considered important, i.e. severity andpotential for health, although others may beadded. So, a question may ask ‘if there are xpeople in adverse health situation A and ypeople in adverse health situation B, and if you can only help (cure) one group, which groupwould you choose?’,250–253 x and y are varied untilthe respondent is indifferent between the two. The two states are then compared with each

§ The search for AHP healthcare articles utilised the strategy detailed in chapter 2 in addition to the Hierarchondatabase of AHP applications on the Internet (www.expertchoice.com/hierarchon), and contact was made with James Dolan of the Unity Health System in Rochester, NY, USA.


12

other in terms of undesirability: B is x/y times asundesirable as A.252,254,255

Willingness to payWTP is a choice-based approach where individualsare presented with a choice between not havingthe commodity being valued, and having the com-modity but forgoing a certain amount of money.The money that they are willing to forgo to havethe commodity is their WTP for that commodity.Using survey techniques, WTP can be estimatedusing four techniques: open-ended (OE); biddinggame; payment card (PC); and closed-ended (CE).Using the OE technique respondents are askeddirectly what the maximum amount of money isthat they would be prepared to pay for a com-modity. If the WTP study is carried out via aninterview, the bidding technique can be used. Here individuals are asked if they would be willingto pay a specified amount. If they answer ‘yes’, the interviewer increases the bid until they reachamounts that the respondents are not willing topay. If they answer ‘no’, the interviewer lowers thebid until they say ‘yes’. WTP is estimated directlyfrom the data provided. A variation on this is thePC technique. Here respondents are presentedwith a range of bids and asked to circle the amountthat represents the most they would be willing topay. An individual’s true maximum WTP will liesomewhere in the interval between the circledamount and the next highest option. The CEapproach asks individuals whether they would paya specified amount for a given commodity, withpossible responses being ‘yes’ or ‘no’. The bidamount is varied across respondents and the only information obtained from each individualrespondent is whether his or her maximum WTP is above or below the bid offered.

WTP has been widely used in healthcare to elicitpublic views.256–258 The application of the WTPtechnique to healthcare has focused on using the PC and CE approaches.259–263

Measure of valueThe measure of value (MoV) technique wasdeveloped by Churchman and Ackoff264 within the context of optimal decision-making generally.Only one study was identified which had appliedthis technique to healthcare.265 The methodidentifies the optimal bundle of services to beprovided within the given resource constraints.Respondents are first asked to rank options.Suppose there are four options ranked as O 1 > O 2

> O 3 > O 4 (where > indicates preferred to). Therespondent then assigns values (v 1…v 4) between 0 and 1 to each option. It is common to assign a

value of 1.00 to the most preferred option, andsuccessive values to the other options which mayreflect strength of preferences. Assume that O 1 =1.00, O 2 = 0.80, O 3 = 0.50 and O 4 = 0.30. Thesevalues are ‘first estimates’. The respondent nowcompares O 1 against the combination O 2 + O 3 +O 4. Assuming he or she still prefers O 1, the valuesattached to each option (i.e. v 1, v 2, v 3, v 4) shouldbe adjusted (if necessary) such that v 1 > v 2 + v 3 +v 4. Given the above values, v 2 + v 3 + v 4 = 1.60 > v 1,the values must be adjusted, for example: v 2 = 0.4,v 3 = 0.25 and v 4 = 0.15. The relative values of O 2,O 3 and O 4 must be retained. O 2 is then comparedto O 3 + O 4. If O 3 > O 4, another adjustment ofvalues is necessary since v 3 + v 4 = 0.40 = v 2.Adjusting v 3 to 0.30 and v 4 to 0.20 results in v 3 + v 4 > v 2. The final values v 1 = 1.00, v 2 = 0.40, v 3 = 0.30 and v 4 = 0.20 are then standardised bydividing them by the sum of v 1 + v 2 + v 3 + v 4. The resulting values v 1 = 0.53, v 2 = 0.21, v 3 = 0.16 and v 4 = 0.10 represent the values or scoresfor each option.265,266 These scores are linked upwith costs, and the optimal bundle or package ischosen from the resources available.

Allocation of pointsA number of techniques have been developed that involve allocating points between alternativeoptions or criteria. Such techniques assume that individuals know the weights they attach todifferent criteria, and can state them. This is incontrast with what has been called ‘policy cap-turing’ approaches, where it is assumed thatindividuals can provide overall evaluations, but that they cannot directly estimate weights forindividual criteria. (Examples of ‘policy capturing’approaches include all CA exercises introducedabove, as well as the various methods for estimatingQALY weights.) One of the earliest applications of this approach was the Hoinville PEM.267,268 Thistechnique was developed to aid town planners intaking account of the preferences of potentialresidents. Potential residents were allocated anumber of points and asked to allocate them to the factors identified as important.

Variations of this method have been used inhealthcare when asking patients to prioritiseaspects of their lives that are most affected by aparticular condition. Most of them are designed to assess change over time and therefore areconstructed differently from measures intended to capture a single assessment. Importantly, allleave it to the respondent to identify the factors to be prioritised. One example of this is thepatient-generated index (PGI). Here respondentsare first asked to state the five most important


13

areas of their life affected by their condition. Asixth area is then defined as ‘all other aspects ofyour life not mentioned above’. Respondents arethen asked to rate how badly affected they are bytheir condition on a scale from 0 to 100, where 0represents the worst possible situation and 100 thebest. Respondents are then given 60 points andasked to spend them on improving the six aspectsof their life. The points they give to the individualareas of their lives represent the relative import-ance of these areas. The individual proportionsallocated to each area are multiplied by the ratingto establish a score for an individual patient’squality of life.269–273 Patient-generated quality of life measures are therefore estimated.

A similar approach was adopted in the develop-ment of the schedule for evaluation of individualquality of life – direct weighting (SEIQoL–DW)technique.274,275 The SEIQoL–DW is an extension of the SEIQoL (introduced above) with a moredirect method for estimating weights. Rather thanusing regression techniques to estimate weightsindirectly, an allocation of points type approach is used to estimate weights directly. Respondentsare presented with a pie chart with a 0–100 scaleround it. Colour segments represent the differentaspects of life, and respondents are asked to adjustthe colours on the chart until each colour reflectsthe relative importance of that area. Quality of life scores are estimated in the same way as for the PGI.

A similar approach has been adopted in thepriority-setting literature, although the techniquehas been referred to as the budget pie#. Thisapproach differs from the PGI and SEIQoL–DW in that respondents are told the areas in whichthey are to allocate points.276–278 This technique,and variations of it, have been applied in a number of settings.276,277,279–286 Initial applicationsconsisted of presenting an individual with a circlerepresenting the budget for public services andasking respondents to cut the pie into piecesrepresenting the amount for various budget items. Individuals were given examples of presentbudget allocations and were given examples of how the budget translated into service levelprovision.277 Within healthcare, Honigsbaum and colleagues,280 in a study concerned with setting priorities in Southampton Health Authority, asked 12 commissioners to allocate 100 points between five criteria used as a basis for

priority setting: health gain; equity; local access;personal responsibility; and choice. A similarapproach was applied in Oregon’s attempt to setpriorities for health services.285,286 Here commis-sioners divided 100 points between three criteria:value to society; value to an individual at risk ofneeding the service; and essential to a healthcarepackage. Seventeen categories of health were then scored against these three criteria on a scale of 1–10.

A variation of the allocation of points approachwas used by Ratcliffe281 within a CA framework.The application was public preferences concerningthe allocation of donor liver grafts. Respondentswere presented with eight choices. Each choicepresented individuals with two groups of patientsthat differed with respect to their age (40, 50 or60), whether or not they had a alcohol-related liver disease, time already spent on the waiting list (3, 6 or 12 months) and whether they hadpreviously had a transplant. For each of the eightchoices respondents were asked to allocate 100 livers between the two groups of patients.

Qualitative techniques foreliciting public viewsThis section concentrates on qualitative techniquesthat have been, or could be, used to elicit publicopinion in healthcare. The methods identifiedhave been classified as individual or group-basedapproaches. Individual approaches included one-to-one interviews and the Delphi technique, andgroup-based included focus groups, citizens’ juries,consensus panels and public meetings. Appendix 3presents a summary of studies identified (andmethodological issues addressed; see chapter 6).

Individual approachesOne-to-one interviewsThe one-to-one interview is a technique based on a researcher (the interviewer) meeting arespondent (the interviewee) on an individualbasis in order to seek the views of the latter.Interviews can be separated into three broadcategories: structured, semi-structured andunstructured. While structured interviewing is very much a quantitative method,9 the latter twoare qualitative. Interviews can be conducted face-to-face, by telephone, or more recently byemail.287 Interviews have been used to investigate

# The budget pie has also been referred to as constant sum measurement, point voting system,562,652 coupon scale andmethod of marks.561


14

a variety of topics within the healthcare setting. For example, Ayanian and colleagues288 looked at the effect of racial differences on patients’preferences and expectations concerning renaltransplantation; Dicker and Armstrong289 investi-gated the underlying assumptions of intervieweeresponses on patients’ views on priority setting;Williams and colleagues290 used individualinterviews as a way of establishing whether and how people evaluate services they receive;Crabtree and Miller291 discussed the long interviewas utilised in a study on pain perception, Wilsonand colleagues292 discussed the merits and pitfallsof using face-to-face and telephone interviews;Groves and Khan (in Baker8) investigated the costsof interviewing; Foster287 looked at the possibility of using email services to conduct interviews; andSohier293 considered dyadic interviews.

The dyadic interviewSohier293 proposed the dyadic interview as amethod where people may find it more acceptableto talk openly about difficult topics. The dyadicinterview is a tool that enables two people to beinterviewed at the same time. The two people areusually related to each other, for example a spouseor parent, or are close friends.

Case study analysisCase studies are gained using standardunstructured interviews and observations, but the way the results are presented is unique. Adescription, along with the researcher’s approachto understanding the case, is presented anddescriptions of the major components of the case are detailed.9

The Delphi techniqueThe Delphi technique obtains attitudes and beliefs about a certain topic from members of aselected panel without the need for them to attenda meeting. Questionnaires or interview schedulesare posted out to individuals; the questions are OE and seek feedback on an individual level. Theinformation gathered is compiled into anotherquestionnaire and sent back to the participatingindividuals who are asked to state their level ofagreement with the statements made. This processis repeated and the rankings analysed to ascertainthe degree of consensus.9 Thus, there are bothquantitative and qualitative elements to thismethod. Examples of where this method has beenused in healthcare include: Guest and colleagues294

who used “interviews with a Delphi panel” togather information on a cost analysis of palliativecare for terminally ill cancer patients; Endacottand colleagues295 used a modified version of

the Delphi technique to gain the opinions ofpaediatric intensive care sisters on the needs ofcritically ill children, and additionally evaluated the method; in Harrington’s study,296 senior prac-titioners were recruited to decide on researchpriorities on occupational health; Charlton andcolleagues297 looked at the spending priorities of different health professionals; Roberts andcolleagues298 compared the responses of con-sultant geriatricians and patients concerningimportant performance measures; Hadorn andHolmes299 used Delphi to ask surgeons, cliniciansand “relevant specialists” their opinions on electivesurgical procedures with a view to formulating a set of criteria to assess the extent of expectedbenefit; Gabbay and Francis300 applied the Delphitechnique to ask general surgeons and anaes-thetists on a national and local level their views on the maximum potential for day surgery; Wilsonand Kerr301 asked bioethics society members andtheir designates about important social valuesrelated to healthcare; Thomson and Ponder302

used the Delphi technique to develop a surveytechnique; Gallagher and colleagues303 and Burnsand colleagues304 involved patients in the Delphiprocess; and finally, Kastein and colleagues305

dealt specifically with the issue of reliability of the technique.

Complaints proceduresComplaints procedures are noted here onlybecause Local Voices mentioned this as a method of incorporating public preferences.1 Using thisapproach, a system is set in place where users of healthcare can register their complaints. These complaints can then be investigated. Hull Health Authority has been cited as using such an approach.1

Group-based approachesFocus groupsIn a focus group a small number of individuals are brought together and encouraged to discussinteractively with other group members, under the guidance of a moderator or facilitator, anumber of specified issues or topics. The inter-action is a crucial part of the process. This tech-nique is commonly used in the field of businessstudies to look at ways in which the problems andopportunities of marketing can be examined.306

It is only in the last decade that it has been used in the social sciences,307 and is now being usedfrequently in health research.308 A focus groupshould be small, usually 8–12 people,306 so as not to be intimidating and so that everybody has the opportunity to express their views.Traditionally there are several rules associated with


15

carrying out focus groups. It is stated by Morgan309

that it is inappropriate to explore complex andsensitive issues, i.e. that the topic of discussionshould be acceptable to participants, and thatgroup members should not know one anotherbefore the interview commences.310 Additionally, it is the task of the moderator to set ground rulesand to ensure that participants remain focused onthe discussion issue.309,310 There are many examplesavailable demonstrating the use of focus groups inthe healthcare setting, and a selection of these arepresented. Two studies that have been used to look specifically at priority-setting issues were theSomerset Health Authority311 and also Kuder andRoeder.312 Cohen and Garrett313 looked at thesensitive subject of client/worker relationships inthe residential mental health setting. Kitzinger314

investigated people’s attitude towards AIDS, whilst Ward and colleagues315 investigated maleattitudes towards vasectomy. Carey and Smith316

investigated psychosocial factors affecting peopleinfected by HIV, using the military as a targetpopulation, and Powell and colleagues317 askedboth users and providers of mental health servicesabout service provision. Wilkinson318 investigatedinteraction between women suffering from breastcancer and Stevens319 looked at experiences oflesbian women in health. Both Keller andcolleagues320 and Smith and West321 explored the healthcare needs of the elderly. Attitudestowards England’s health strategy were investi-gated by Bradley and colleagues;322 Dolan andcolleagues323 looked at how much people’s views changed after deliberation; and Ramirez and Shepperd324 explored attitudes towardsdifferent risk factors associated with cancer. The final example is provided by Weinberger and colleagues325 who were concerned withresearcher consistencies when rating transcripts.

Concept mappingThis method has been suggested by Southern and colleagues326 as an alternative to using focusgroups. It is very similar but ensures that all mem-bers have an equal opportunity to express opinionby eliminating the problems that group dynamicscan cause. Additionally, it uses a combination of qualitative and quantitative data collectionmethods. A series of maps are developed to estab-lish links between ideas and suggestions expressed.There is feedback provided to the participantsduring the process. This provides an opportunityfor respondent validation, therefore increasing the overall validity of the process. Trochim andLinton327 suggest that this is an appropriateframework for decision-making, and therefore maybe particularly useful for setting priorities.

Citizens’ juriesCitizens’ juries consist of a group of people,representing the lay population, which discussesissues on the basis of evidence provided by experts.Citizens’ juries usually occur over a period of 4 days and consist of a small group of 12–16328 or10–20 members.329 Participants are selected usingrandom and stratified sampling to be as represent-ative of their community as possible. They shouldinclude one329 or two trained moderators.328 Theyinvolve members of the public in decision-making,rather than simply asking their opinion, thus taking the process a step further. The jurors arebriefed about the topic in question, are givenwritten information and listen to evidence fromwitnesses. Information presented to the jurors must come from several points of view and bepresented in a fair way.330 This is normally achievedby using witnesses to illustrate their cases. Themembers are then given the opportunity to cross-examine the witnesses and are able to discussaspects of the subject in smaller groups. A mod-erator should be present for these discussions, butmay not be present for the final decisions;330 thejurors’ verdict does not need to be unanimous and is not binding. A report is produced whichdescribes the deliberations and conclusions/recommendations of the jurors and is submitted(after the jurors have had the chance to examineit). Five pilot studies were conducted in 1996,328

and during this series the method was evaluated for effectiveness of gathering the public’s opinionson priority setting328,331,332 and specific health issueswere debated, namely mental health and palliativecare issues as well as payment methods for healthservices.331 Additionally, three juries were held byThe King’s Fund, which dealt with issues concern-ing gynaecological cancer, back pain and whoshould perform certain treatments.333 Also, theWelsh Institute for Health and Social Care(WIHSC) used this technique to explore newgenetic technologies;334 juries were also held inPortsmouth and Nottingham328 and Lewisham.335

Consensus panelsThese are a simplified version of citizens’ juries,and consist of small groups brought together to discuss a particular issue. These groups areprovided with limited information on specificscenarios. The participants are then asked to give reasoning behind the choices that they have made. Lengthy discussions then take placeand are led by one of the panel members, andobserved by a researcher. This technique has been applied to many different health themes,including the opinions of patients, the generalpublic, general practitioners (GPs), specialists


16

and health insurers to cuts in healthcare expendi-ture,336 birthing centre provision,337 reasons for theunder-treatment of depression,338 synthesis of bestpractice for health technology assessment339 andtreatment for lower back pain.340

Public meetingsPublic meetings are a common way to gain publicopinion relating to health service issues. Plans and proposals are sent to CHCs for comment, and meetings are arranged to obtain a feel forattitudes towards these.329 Many health authoritiesuse public meetings as a means to provide inform-ation to communities and to gauge public opinionon health issues. Gundry and Heberlein341 investi-gated the types of people that attend publicmeetings in order to test their representativenessand Broadbent342 investigated those issuessurrounding advertisement and access to publicmeetings in her local area. Finally, Gott andWarren343 investigated a neighbourhood healthforum that was interested in increasing localparticipation in health issues.

Nominal group techniqueA consensus method that is similar to the Delphitechnique is the nominal group technique. Thiswas developed to avoid the problems of groupinteraction and provides a quantitative measure of qualitative data. In one meeting ideas can begenerated and problems solved.344 It takes the form of structured meetings facilitated by a thirdparty and major issues affecting a particular groupare identified and ranked. This method is stated as being appropriate for prioritising interventionsand effective at obtaining consensus.345 There are a number of steps in this highly structured process.Participants in the group write down their ownideas, the ideas of each participant are then listedone by one, and these suggestions are discussedand grouped into clusters. Each participant thenranks the ideas in order of importance, the rankingis discussed and ideas are re-ranked as a group.Feedback of the final results is provided.346–348

Redman and colleagues349 investigated use of thenominal group technique to look at priorities in

breast cancer service provision in Australia. A rangeof health professionals, patients and relatives ofpatients were invited to attend the workshops: 274agreed to participate. Care was taken to includewomen who were non-English speaking, and theymade an important contribution to the study.

Discussion and conclusions

A number of quantitative and qualitativetechniques have been identified. Whilst some arewell established in terms of their use in healthcare,other new approaches have also been identified.Some of these seem to be particularly applicablefor measuring public preferences.

A number of quantitative techniques wereidentified which have been widely used in health-care. These include simple ranking exercises,rating exercises, satisfaction surveys and methodsfor estimating quality weights within the QALYparadigm (visual analogue, SG and TTO). CA(ranking, rating and discrete choices) and WTPare being developed within the context of health-care. Techniques with limited applications inhealthcare include the QDP, MoV, AHP andallocation of points.

Of the qualitative techniques, one-to-one inter-views and focus groups have been widely used in many different fields and more recently inhealthcare, with a current focus on their uses inpriority setting. Other methods such as citizens’juries are relatively new, having been recentlydeveloped in the context of decision-making. In addition, whilst case study analysis, conceptmapping and nominal group techniques are in the early stages of development, they presentunique ways of analysing data collected in aqualitative format.

In the next chapter, the criteria by whichtechniques should be addressed is discussed, andin chapters 5 and 6 the techniques identified arereviewed according to these predefined criteria.


17

In this chapter we attempt to summarise thecriteria that has been used to evaluate the

quality of quantitative and qualitative research.This was done by identifying both the methodo-logical issues addressed in the papers identified in the systematic literature review (chapter 3) and current debates in the literature concerningappropriate criteria with which to judge tech-niques. From this the project group decided on a set of criteria for evaluating the techniquesidentified in chapter 3. Given the different natureof the techniques used, whilst an attempt was made to evaluate quantitative and qualitative by the same criteria, the final criteria were slightly different.

Criteria for assessing quantitative techniquesKey references used, in addition to articlesidentified in the review, were Brazier andcolleagues253 from their recent review of healthstatus measures in economic evaluation, andStreiner and Norman350 from their book ondeveloping and using health measurement scales.The quantitative techniques identified in thesystematic review of methods for eliciting publicpreferences, together with these two references,identified a number of criteria that have been used to assess the value of techniques. It should be noted here that different disciplines useddifferent definitions of criteria, and we haveattempted to provide an overview. Some readersmay therefore disagree with our definitions.However, we hope that we have covered the main criteria.

ValidityA common criterion mentioned in a number ofstudies was that of the validity of the techniqueused. A valid measurement is one that measureswhat it is supposed to measure.351 Three types ofvalidity were identified: content validity, criterionvalidity and construct validity.352

• Content validity refers to the extent to which a measure takes account of all things deemedimportant in the construct’s domain.

• Criterion validity (or external validity) isconcerned with whether the measure adoptedmeasures what the researcher is trying to measure.

• Construct validity – there are two types:convergent validity, which measures the extentto which the results are consistent with othermeasures that are held to measure the sameconstruct; and theoretical validity, which assessesthe extent to which the results are consistentwith a priori expectations.

Reference has also been made in the literature tofactors such as framing effects353–356 and strategicbiases,253,257 the presence of which may cast doubton the overall validity of the technique.

Reproducibility *

This is defined as repeatability of results over agiven time. This is usually assessed as test–retestreliability whereby a sample of respondents repeatthe same exercise after a short time period andtheir results are compared with those first timearound. For continuous variables this is measuredusing a correlation coefficient (r),46,253 whilst fordiscrete responses the Kappa coefficient (κ) isused.357 It should be noted that implicit in thismeasure of reliability is an assumption thatpreferences exist (are complete) and are stableover time. If these assumptions are violated (which may well be the case in healthcare, whereindividuals are not used to making choices), thenpoor reliability may be indicative not of a poortechnique but rather of the fact that respondentsdo not have complete or stable preferences. This is discussed in more detail in chapter 8.

Internal consistencyA number of studies were identified which hadaddressed the issue of internal consistency. Thisrefers to the extent to which respondents haveanswered questions in a logical manner. The

Chapter 4

Criteria for assessing the methodological status of techniques

* Inconsistency and reproducibility are consistent with Fitzpatrick and colleagues’388 definition of reliability.


18

articles identified in the literature review indicatedthat this has been tested in a number of ways inthe literature:

• One common test for consistency of responses is to include dominant options, i.e. if two health-care interventions/services are being valued,and one is obviously superior to another, thenthe superior one should have a higherweight/score/rank.61,196,197,199,203,205,208,253,358–365

• Tests were identified in the literature wherepreferences elicited using one technique were then compared to those elicited using a second technique to see if they gave consistent results.262,355,366–369

• Some techniques test for consistency withrespect to the stability of preferences within atask. Here individuals are presented with thesame choices/items/tasks within a survey. Ajudgement reliability coefficient measures theextent to which responses are correlated.67–70

• Alternatively, attitude-type surveys often include statements intended to measure thesame construct, and then tests are conducted to see if responses move in the same direction.This measure is usually expressed as Cronbach’salpha (α),80,87–91,167,173,176–178,180,182,370 the coefficientof reproducibility95,98,102 and the coefficient of scalability.95,102

• One test of consistency used in surveys is that of transitivity. This states that if there are threeprogrammes being valued (A, B and C), and arespondent states that they prefer A to B and Bto C, then they should also prefer A to C.193

Acceptability to respondentsAcceptability to respondents was identified in the literature as an important criterion. Theargument posed here is that the questions posedby the survey technique should be realistic, suchthat respondents can answer them in a way whichreflects their true preferences. The acceptability ofthe technique may be assessed by factors such asthe questions posed, time to complete, responserates and completion rates†.

CostSome studies made reference to the cost of thetechnique. Cost is defined broadly to include thefinancial, administrative and analytical burden ofthe technique to users. The costs to the researchers(or the commissioners of research) can be splitinto administrative resource use and analytical

resource use. The reference to this criterionreflects the fact that whilst cost does not indicateanything about methodological quality, it repre-sents a potential barrier to use of techniques, i.e.individuals concerned with eliciting public prefer-ences may not use a CA exercise because they donot have the necessary expertise to do this.

Criteria for assessing qualitativetechniquesWhilst it is generally accepted that quantitativeresearch can be evaluated according to a set ofpredefined criteria, and there appeared to be some consensus in the literature about what thesecriteria were, although the terminology variedacross disciplines, it became clear from thequalitative literature that there are conflictingtheories/approaches on the optimal way toevaluate qualitative work.13 Four general views were identified:

All research perspectives are uniqueand each is valid in its own termsThis view argues that producing lists of ‘universal’quality criteria is unrealistic for qualitative analysis.They are seen as being restrictive and producingsanitised results.13,371,372 According to this view,when using quality criteria or checklists there is the risk of producing a general consensus ratherthan true findings. This in turn reduces the valueof the research. Here it is advocated that judge-ment be used with each piece of work and that the method of data collection is the most import-ant criterion. It is stated that the data yielded must be taken in the correct context and that how the data are gathered is crucial, including the role of the researcher. This view rejects thenotion of establishing quality criteria and, there-fore, has little support from applied health(service) researchers,373 commissioners and research funders.

Qualitative and quantitative methodsboth can be assessed but not by thesame criteriaThe second view advocates that it is possible to assess the quality of qualitative techniques by predefined criteria, but different criteria to quantitative must be used. Lincoln and Guba374 suggest four criteria for evaluatingqualitative research:

† It is recognised that such factors may not be driven so much by the nature of the technique but rather by factorssuch as complexity of scenarios presented and question framing.


19

• credibility – which refers to some kind ofendorsement of the research findings by theresearch participants

• transferability – similarities exist betweendifferent settings and it is possible to developworking hypotheses which have potential forsome transfer between settings

• consistency or dependability – ability to trackfindings to their source

• confirmability – providing an audit trail to allow other researchers to examine the processwhereby the original researchers arrived at their results‡.

Whilst these criteria have been argued to bedifferent to those employed by quantitativeresearchers, they have parallels with validity,generalisability, reliability and objectivity/neutrality, respectively.

Qualitative and quantitative methodscan be assessed by the same (or verysimilar) criteriaThe third view is that qualitative and quantitativeresearch are different approaches to representingthe same reality. Therefore it is possible to judge the quality of both sets of research methods against a common standard, although theassessment may have to be modified to take intoaccount the practical differences between the twoapproaches.13,371,375 The majority view indicates thatthe main criteria to ensure quality are: acceptability;cost; validity; reliability; generalisability (or rele-vance) and objectivity. Most of these criteria applyalso to the valuation of quantitative techniques andwere defined above. However, it is recognised thatsome of these criteria require some modificationwhen applied to qualitative research.371

AcceptabilityAcceptability as a criterion is important in a culturethat is dominated by statistics. Acceptability ofqualitative methods amongst users, policy-makersand managers will be an important factor in thedecision on which method to use.

CostUsing qualitative methods can be expensive but,despite this, cost is not generally included in thelist of qualitative criteria. Choosing a methodmerely because it is of low cost is not acceptable.For example, if the only reason for carrying out afocus group instead of individual interviews is

because it will be cheaper, then that money can beseen as wasted if it compromises the quality of theresearch outcome. What is esssential is that, inassessing all types of research methods, there issome estimate of the likely value for money: will a more costly method produce even more useful findings?

ValidityValidity was defined above as measuring what you intend to measure.351 Within the qualitativeliterature a number of approaches have beenproposed to assess validity:

TriangulationTriangulation originally referred to the use ofthree or more different research methods. Itsdevelopment was a response to the recognisedneed to guard against errors in validity by testing whether consistent findings emerge from different methods.7,9,10,376,377 This mixing and matching of methods has also been referred to as methodological pluralism and multipleresearch approaches. Whilst it originally meantusing three methods at the same time, it ispresently used more loosely for any study usingmore than one technique, on the same population,addressing more or less the same issue. Twoapproaches have been adopted in the literature:

• simultaneous triangulation – refers to using a number of techniques at the same time, on the same population, addressing the sameissue, and seeing if they come to the sameconclusions; to this extent it can be viewed as equivalent to convergent validity within the quantitative criteria

• sequential triangulation – possibility of usingdifferent methods sequentially, such that each provides a basis for the development ofsubsequent stages of the research process.377

Denzin11 has expanded on this and has identifiedfour types of triangulation: (i) data triangulation;(ii) investigator triangulation; (iii) theory tri-angulation; and (iv) methodological triangulation.These take triangulation beyond merely a methodof analysis and validation of results and highlightits use as a method of enriching the researchthroughout the entire process.10

Miller and Glassner (in Silverman378) used themetaphor of a bridging method to describe a way

‡ These definitions are taken from Lincoln and Guba.374 It is recognised by the authors that the definition ofconsistency appears the same as that for confirmability.


20

of forming links between different types of analysis.She focused upon “using several methodologicalstrategies to link aspects of different sociologicalperspectives, not to discover indisputable factsabout a single social reality”, something that shesees triangulation as doing.

It should be noted that problems identified in the quantitative literature concerning convergentvalidity have also been raised within the qualitativeliterature. For example, if a number of methodsprovide contradictory results, how is it decidedwhich results are to be used?379 Similarly, if all thedata gathered using different research methodslead to the same conclusion, how can we know that these are ‘right’? It is possible that anothermethod still would produce different results.375

Further, triangulation is often seen as an attempt to make the research process more scientific but, in doing this, one might lose the personal perspec-tive.11 Murphy and colleagues reject triangulation,concluding that “a clear exposition of the processof data collection and analysis, in which the dataare related to the circumstances of their pro-duction, is essential to the evaluation of findingsfrom qualitative research”.371 Thus, the importanceof the validity of each methodological processrather than the need to replicate findings bydifferent methods appears to be the dominantexpert view of the contribution of triangulation.

Respondent validationRespondent validation involves asking participantswhether they agree with the findings of the study.If they agree then support is provided for thevalidity of the technique. If they do not agree, new refinements may be made to existing methods.Such a method of validation could also be appliedin quantitative research. This approach has beenproposed as the best way to test the validity of aqualitative study.374 However, MacPherson andWilliamson point out possible pitfalls of thistechnique, such as the fact that respondents’ views and comments may not be easily incorpor-ated. They recommend that “respondents in anyvalidation exercise have a responsibility to abide by the agreed ‘rules’ of the process, or decline to take part, stating why, and accept their position being recorded in the final report”.380

Reflexivity§

Reflexivity is a process where researcherscontinually reflect upon how their own interests

and potential biases could alter the interpretationof the results.308,371 Researchers come from differ-ent backgrounds, cultures and have different ideas,all of which may affect the types of questionsposed. It is therefore useful to the reader of quali-tative studies if the investigator describes back-ground and training so that he or she can decidehow much these factors affect the piece ofresearch. The underlying concept is that noresearcher is neutral or totally objective and there-fore a reader should attempt to ascertain their rolein the research process.372 Mason strongly advo-cates the use of this technique in every decisionmade by researchers in order to strengthen thetrustworthiness of the qualitative method.372

Additionally, Banister and colleagues suggestedthat continuous discussion with colleagues is bothstimulating and challenging and aids in extendingunderstanding and clarity.381 Also, in order to makeexplicit how ideas were formed, the keeping of adetailed diary is extremely valuable. In this way thereader of the piece of work can judge for himselfor herself the context in which an assumption wasmade and, in turn, how valid that assumption is.

Again, this type of approach may be useful inquantitative research that is carried out in aninterview setting.

Data analysisData analysis of qualitative data is as vulnerable toaccusations of variability as are its methods of datacollection, and it is vital that the method of dataanalysis be sound. The research will be seriouslyflawed if the data are not analysed in a systematicand rigorous way, regardless of how carefully thedata were gathered. Miller and Glassner (inSilverman378) state that there is always a problemwith re-telling an account of an event because it is fragmented and could be taken out of context.The process of analysis, coding and categorising,for example, means that only parts of accounts are told and cannot be viewed by the reader as awhole. The meanings of what the respondent saysmay be further misinterpreted by the readerhim/herself. This is particularly true if the readerdoes not belong to the same primary group asthose being interviewed, where social or culturaldifferences are evident or where language use isunconventional. Qualitative research has beencriticised for producing vast quantities of datalacking in structure.382 There are a number oftechniques that may help to guard against theproblems of data analysis.

§ Reflexivity is relevant to both validity and reliability.


21

A number of computer packages have beendesigned to aid the analysis of qualitative data. As qualitative research tends to produce largevolumes of data, this is particularly helpful inreducing the time taken for analysis. However, the use of computer packages has been criticisedfor being impersonal. A computer has no ‘feel’ for the results and may therefore produce lessinteresting results.308 There may also be a problemwith loss of data due to computer error, and it may introduce problems with confidentiality asthere is the potential for more people to haveaccess to the data set.383 Nonetheless, it is a trendwhich is likely to continue, and there is a need toassess critically the benefits and costs of new pack-ages as they appear, and the comparative advan-tage, especially for small studies, of computerisedversus manual analyses. An important point to notehere is that, whilst quantitative research is usuallyanalysed using computer packages, problems stillarise since different methods of analysis may givedifferent results (e.g. probit analysis versus logitanalysis of discrete variables).

Qualitative data analysis can be done by methodsother than qualitative computer software packages.One example of this is the framework method,which has been developed by Ritchie and col-leagues (personal communication) at the NationalCentre for Social Research, London. This methodof analysis is worth considering.

Grounded theory is a process that reveals theoryfrom data collected.9,384 This “is a style of doingqualitative analysis that includes a number of dis-tinct features, such as theoretical sampling, andcertain methodological guidelines, such as themaking of constant comparisons and the use of acoding paradigm, to ensure conceptual develop-ment and density”.385 The emphasis is on organisingin a formalised manner the ideas that are putforward by the participants. A strength of thismethod is that the researcher can shift the focus of the study as the data are collected. It is often thecase in qualitative work that the outcome will beunknown at the start, so the direction it takes maychange in accordance with the kinds of informationgathered. Bowling argued that if this technique isused “the data are collected and theories andpotential concepts and categories are developedduring the process, more data are collected and the theories, concepts and categories tested and soon until an understanding of the phenomenon isachieved”.9 This is a widely accepted method ofanalysis and serves to improve the rigour of aqualitative piece of research. It may also be useful in the area of quantitative research.

Analytic induction has also been proposed within the context of qualitative research. The aim of this type of evaluation is to identify ‘deviant’ cases, and to move the theory forwarduntil it can explain all or most of the subjectsbeing studied.10,386 This technique begins with arough idea or definition of a problem. The datacollection process begins on one or two cases and then the original definition is reassessed,taking into account what has been realised fromthe cases examined. More cases are studied untilthere is a correlation between the cases and thetheory. Deviant cases must be identified and thecyclic process repeated until all cases fit the theory.This is a very demanding and time-consumingform of analysis but is recognised as adding valueand rigour to qualitative research.308 This approach can be seen to be similar to the internal validityapproach in quantitative research.

ReliabilityAs stated above within the context of quantitativeresearch, a reliable measurement is one whichwhen repeated gives a similar result on eachoccasion.9,10,351,387 Within the qualitative literature,reliability is often presented as a combination ofreproducibility (test–retest reliability) and internalconsistency/transitivity.388 However, Kirk and Miller define three types of reliability relating to qualitative research:389

• quixotic reliability – examines how far themethod can continuously lead to the sameresults (which may be defined as test–retest reliability)

• diachronic reliability – the stability of theobservation over a period of time (if this period of time is within the same question-naire, this may be defined as internalconsistency as measured by the judgementcorrelation coefficient; if this period of time is across points in time this may be defined as test–retest reliability

• synchronic reliability – the consistency of resultsgathered at the same moment in time but byusing different techniques (which may bedefined as convergent validity or sequentialtriangulation).

Methods to improve reliability have been put forward by Kirk and Miller.389 When datagathering, it is advisable during note-taking tostandardise the method of recording and agree on, for example, a number of punctuation signs.To avoid confusion during analysis, the ideas from the observed and the observer (researcher)should be noted separately.


22

GeneralisabilityThe criterion of generalisability is concerned with whether wider claims can be made on thebasis of the research.372 Generalisability theoryrecognises that in any research context there areinfinite sources of error. In quantitative research,sampling and statistical techniques exist to correctfor possible errors.350 For this reason it did notfeature as a main criterion in the quantitativeliterature. Such approaches may be more difficultto adopt in qualitative research. An alternativeapproach, more commonly employed in qualitativeresearch, is theoretical sampling. Here the samplechosen for the study is held to be representative of the factors that may explain variations in thesubject matter being studied. If a grounded theoryapproach is adopted then the study sample may beextended as the study proceeds. However, othersargue that once an attempt is made to generalisefindings, the context is lost and it is very difficult to state whether the findings would be applicablein another settings.10 Lincoln and Guba stipulatedthat generalisations spanning different contexts are impossible but suggest that conclusions drawn can be comparable and transferable across different settings,374 provided that thesettings are described and understood.

ObjectivityObjectivity has been argued by some researchers to be a fundamental part of the research process,8,9

although other qualitative researchers have dis-puted this, arguing that it is a highly contentiousepistemological position#. Using qualitativemethods, the process and outcomes of the research can be seriously affected by the priorknowledge, experience and opinion of theresearcher. Indeed, it is impossible to achieve true objectivity; any professional will enter into a project with preconceptions of what the out-comes may be, and to remain entirely neutral and objective is an unrealistic goal.9 There aremethods that can be employed to maximiseobjectivity: these are reflexivity (see page 20) and intersubjectivity.

Intersubjectivity is an approach whereby one ormore additional researchers are brought in toassess whether or not they achieve similar findingsfrom the data set as does the original researcher.386

In order to achieve intersubjectivity (which is, ofcourse, not the same as objectivity) in the analysisof qualitative data, more than one researcherneeds to code and analyse the content of a

particular data set. Each must be allowed completefreedom to select the different categories foranalysis. After going through the same contentanalysis procedure researchers can then comparenotes. Discrepancies in the coding need to bechecked and negotiated until an agreement isreached. Thus, the coders are developing codingschemes independent of each other. The codingcategories were developed as the coding pro-gresses, starting with the first document of theseries. To some extent this still leaves open theissue of objectivity of the actual content analysis, in ways similar to the debate about triangulation.More than one researcher analysing the same data set reduces the subjective element of a study; in other words, it may lead to a higher level of objectivity.

It should be noted that such an approach mightalso be used in quantitative research, whereinterviews are being used.

Checklists of good practice (instead of formal criteria)Finally, an alternative approach to usingpredefined criteria in assessing the quality ofqualitative techniques is to use checklists of goodpractice. It is important to note that such checklistsfocus on how the research was conducted, ratherthan on the quality of the technique itself. As such,they may be seen as vehicles to ensure validity,reliability, generalisability and objectivity. Withinquantitative research such checklists have not beenwidely used, although guidelines were identifiedwithin the context of conducting WTP studies390–392

and carrying out economic evaluations moregenerally.393 However, within the context ofqualitative research such guidelines have beenargued to be valuable in establishing quality as long as they are used in an “open and permissive way”.375 A number of checklists havebeen created for the purposes of evaluatingqualitative research.13,394,395 Three such lists are shown in Boxes 1–3.

The critical appraisal skills programme (CASP), a teaching resource for evidence-based health-care, formulated ten questions for the appraisal of qualitative methods.394 Mays and Pope395

published a summary of the principal approachesto evaluation in general in 1995, and updated this very recently.13 One of the differences be-tween these two lists proposed by Mays and Pope is that the latter set is wider on the general

# We thank one of the anonymous referees for pointing this out to us.


23

worth of the research (e.g. Box 3, point 1), whilstthe first list is more methodological.

Discussion and conclusion

Following the criteria identified in the quantitativeliterature, the following five criteria were used toevaluate quantitative techniques:• validity

BOX 1 Checklist for qualitative methods – critical appraisal skills programme (CASP)394

1. Was there a clear statement of the research aims?

2. Is a qualitative methodology appropriate?

3. Was the sampling strategy appropriate to addressthe aims?

4. Were the data collected in a way that addressesthe research issue?

5. Was the data analysis sufficiently rigorous?

6. Has the relationship between researchers andparticipants been adequately considered?

7. Is there a clear statement of the findings?

8. Do the researchers indicate links between datapresented and their own findings on what thedata contain?

9. Are the findings of this study transferable to awider population?

10. How relevant is the research?

BOX 2 Checklist for qualitative methods (Mays & Pope, 1995395)

1. Overall, did the researcher make explicit in theaccount the theoretical framework and methodsused at every stage of the research?

2. Was the context clearly described?

3. Was the sampling strategy clearly described and justified?

4. Was the sampling strategy theoreticallycomprehensive to ensure the generalisability of the conceptual analyses (diverse range ofindividuals and settings, for example)?

5. How was the fieldwork undertaken? Was itdescribed in detail?

6. Could the evidence (fieldwork notes, interviewtranscripts, recordings, documentary analysis, etc)be inspected independently by others?; if relevant, could the process of transcription be independently inspected?

7. Were the procedures for data analysis clearlydescribed and theoretically justified? Did theyrelate to the original research questions? How werethemes and concepts identified from the data?

8. Was the analysis repeated by more than oneresearcher to ensure reliability?

9. Did the investigator make use of quantitativeevidence to test qualitative conclusions where appropriate?

10. Did the investigator give evidence of seeking outobservations that might have contradicted ormodified the analysis?

11. Was sufficient of the original evidence presentedsystematically in the written account to satisfy the sceptical reader of the relation between the interpretation and the evidence (e.g. werequotations numbered and sources given)?

BOX 3 Checklist for qualitative methods (Mays & Pope, 200013)

1. Worth or relevance: was this piece of work worth doing at all? Has it contributed usefully to knowledge?

2. Clarity of research question: if not at the outset ofthe study, by the end of the research process wasthe research question clear? Was the researcherable to set aside his or her researchpreconceptions?

3. Appropriateness of the design to the question:would a different method have been moreappropriate? For example, if a causal hypothesiswas being tested, was a qualitative approach really appropriate?

4. Context: was the context of setting adequatelydescribed so that the reader could relate thefindings to other settings?

5. Sampling: did the sample include the full range of possible cases or settings so that theconceptual rather than statistical generalisationscould be made (i.e., more than conveniencesampling)? If appropriate, were efforts made toobtain data that might contradict or modify theanalysis by extending the sample (e.g. to adifferent type of area)?

6. Data collection and analysis: were the datacollection and analysis procedures systematic? Was an ‘audit trail’ provided such that someoneelse could repeat each stage, including theanalysis? How well did the analysis succeed inincorporating all the observations? To what extentdid the analysis develop concepts and categoriescapable of explaining key processes or respon-dents’ accounts or observations? Was it possible to follow the iteration between data and theexplanations for the data (theory)? Did theresearcher search for disconfirming cases?

7. Reflexivity of the account: did the researcher self-consciously assess the likely impact of themethods used on the data obtained? Were suffi-cient data included in the reports of the study toprovide sufficient evidence for readers to assesswhether analytical criteria had been met?


24

• reproducibility• internal consistency• acceptability to respondents• cost.

In addition, three additional properties oftechniques were thought to be important to the research team: (i) whether the technique has a theoretical basis; (ii) involves a constrainedchoice; and (iii) results in a measure of strength of preference. These additional criteria may beseen as properties of the technique itself, and will not be influenced by the nature of the studycarried out. Theoretical basis refers to the extentto which the technique has some theory behind itso that the axioms on which it is based can betested. Constrained choice refers to the extent towhich the technique takes account of the realdecision-making context facing decision-makers,therefore incorporating the concepts of scarcityand opportunity cost. Strength of preference isinterpreted as whether the tech-nique indicatesintensity of preference between choices; in otherwords, whether the technique gives a cardinalrather than ordinal ranking of options¶.

The issue of quality in qualitative research hasclearly received increased attention in recent years. However, there seem to be no set guidelines,with different researchers adopting differentapproaches. Four approaches were identified. The first view, that such research cannot beevaluated, is unlikely to receive much support

from methodologists or funders of research. The second and third views suggest that criteriacan be applied, but that these criteria may beeither different to those for quantitative, or thesame, but operationalised in a different manner. In looking at the second and third views it is clearthat many of the criteria used have some parallelswith quantitative criteria. For example, simultan-eous triangulation is very similar to convergentvalidity, and analytic induction has close ties withinternal validity. Quantitative research may alsogain by extending definitions of validity to includerespondent validation and reflexivity, and appli-cations of quantitative techniques may beimproved by the development of checklists forgood practice. Finally, checklists have also beenproposed as a method for evaluating qualitativeresearch. We adopted the third view. Based on theliterature and discussion with colleagues, sixcriteria were decided upon to evaluate thequalitative techniques identified:

• validity• reliability• acceptability• cost**

• generalisability • objectivity.

Chapter 5 presents the review of the quantitativetechniques identified in chapter 3 and chapter 6the review of the qualitative techniques identifiedin chapter 3.

¶ In an attempt to subject the criteria to peer review to ensure that the criteria being used to assess instruments wereappropriate, a questionnaire was administered to a convenience sample of 67 researchers with some knowledge ofmethodology and who may be involved in eliciting public preferences. The intention here was to assess whether theinitial set of criteria covered all factors that were important to users of research instruments. The sample was takenfrom staff at the University of Aberdeen (including members of the Departments of Public Health, General Practice,Sociology, and Education, and the Health Economics Research Unit and Health Services Research Unit), staff ofGrampian Health Board (Acute Services Team and Primary Care Unit) and members of the Health Economic Study Group (at their bi-annual conference). The results indicated that the criteria chosen were all judged to be important, and no additional criteria were mentioned.** It is noted here that cost is not an evaluation criteria per se but is an important consideration and component ofwhether a particular technique is used.


25

In what follows, the quantitative techniquesidentified in chapter 3 are reviewed according

to the criteria in chapter 4. The presentation isqualitative in nature. Appendix 4 presents aninitial attempt to provide a quantitative synthesis by providing weights for the different criteria. This will be developed in future work.

An important distinction that should be madewhen evaluating quantitative techniques is thatbetween an approach to and an instrument for eliciting preferences. This is an importantdistinction when considering the extent to whichconclusions regarding whether techniques satisfycertain criteria are generalisable. For techniquessuch as SERVQUAL, SEIQoL and generic satisfac-tion surveys it is possible to establish the methodo-logical status of the instrument generally within the patient group and context explored. This isbecause of the instruments’ fixed content anddesign. However, for general approaches to elicitingpreferences, such as CA or WTP, satisfaction of thecriteria will depend to a large extent on the way the research is conducted. Given this, whilst it ispossible to get some overall feel for the satisfactionof the criteria for the approaches, such conclusionsare not as strong as those conclusions made regard-ing specific instruments*. Further, whilst the criteriaof validity, reproducibility, internal consistency,acceptability to public and cost will be determinedby the way the research is conducted, the criteria of theoretical basis, constrained choice andstrength of preference can be seen as properties of approaches and instruments and therefore will not be influenced by the research.

In the process of the evaluation exercise it becameclear that four of the techniques identified in our review (visual analogue, SG, TTO and PTO)had been extensively reviewed by Brazier andcolleagues in their recent HTA monograph.253

We obtained the references listed in the Brazierreport, reviewed them in the light of our criteria,and updated the literature identified by Brazier

and colleagues with studies and issues raised fromour literature review which had not been identifiedby Brazier and colleagues.

Ranking techniques

Approaches to eliciting preferencesSimple ranking exerciseThere is no well-defined theoretical basis forsimple ranking exercises, and the output of suchstudies is clearly ordinal. Whilst the types ofquestions posed may be argued to be relativelysimple (compared with some techniques that will be introduced later), the usefulness of theoutput of such questions to addressing issues ofresource allocation has been challenged.396 Themain limitation is that it is not clear what decision-making rule should be implemented from theresults of a ranking exercise. For example, in theBritish Medical Association (BMA) study, reportedby Groves,17 from a list of ten healthcare inter-ventions, the public ranked childhood immunis-ation at the top of the list, followed by GP care for everyday illness, screening for breast cancer,intensive care for premature babies, heart trans-plant operations, support for carers of elderlypeople, hip replacements for elderly people, anti-smoking education for children, treatment for schizophrenia and finally cancer treatment for smokers. Therefore, should all children beimmunised? And then GP care be provided for all everyday illnesses? And then we move down tothe third priority – should all women be screenedfor breast cancer? From this example the obviouslimitations of a simple ranking exercise are clear.The main problems relate to a lack of consider-ation of the marginal context of decision-making,the lack of consideration to the principle ofopportunity cost, and the failure of the techniqueto provide a measure of strength of preferences.17

Bowling and colleagues16 highlighted the high cost of interview. Potential costs of studies werecited to range from £20,000 to £90,000. The

Chapter 5

A review of quantitative techniques for eliciting public views

* We would like to thank an anonymous referee for making this point.


26

authors further state they did not receive thecooperation of health councils who protested to the ethics of the study (this may have happened for any of the techniques used).

Response rates have generally been good forinterviews,16,18,24–26,366 but more varied for postalquestionnaires.16,22,366 Bowling and colleagues16

reported a response rate of only 11% for a public sample, after four mailings, and 66–68% for doctors, again after four mailings, whilstKinnunen and colleagues22 report postal responserates of 52–68%, including 59% for their generalpublic sample. Bowling,366 in a sample of 350respondents, found item response to vary between 290 and 350.

A limited number of studies were identified that had addressed internal consistency or repro-ducibility. Shickle358 found evidence of internalinconsistency in responses: whilst respondents gave their highest rankings to life-saving treat-ment, they stated a preference for quality of lifeimproving treatments. Bowling366 found identicalrankings in test–retest reliability within their pilotsamples, but could not compare with the mainsample because of changes in wording.

A number of studies have, however, addressedvarious aspects of validity. Several studies havesupported internal validity through confirminghypotheses that respondents favour the young over the old, females over males, non-smokers over smokers, non-drinkers over drinkers, andpoor people over rich.17,18,25,26,358 Furnham andBriggs26 further found that older people would be more likely to discriminate over place of birthand that women would be more likely to prioritisethose with dependants. Bowling,16 in a priorityranking of 12 health services, observed that thepriority ranking of both their public samples(community groups and a postal survey) wereidentical or very similar in the majority of cases.They further found GPs’ and consultants’ rankingsalso to be very similar to each other. However, the authors’ pilot demonstrated that responseswere sensitive to question wording. For example,one of the health services, “intensive care forpremature babies” was given the highest priority(rank 1) but when it was qualified with thestatement “weighing less than one and a halfpounds and unlikely to survive” it dropped to rank 10. Concerning convergent validity,Angermeyer and colleagues24 found a rankingexercise led to more prominent differences inrecommendations concerning schizophrenia than a Likert scale.

Qualitative discriminant processThe QDP is in the early stages of development,there have been no applications eliciting publicviews in healthcare, and little methodological work has been carried out. However, within thecontext of healthcare, the technique does haveappeal. Bryson and colleagues28 note that one of the significant advances of the technique over other methods of eliciting preferences is its ability to deal with the vagueness that exists in human decision-making. This may be partic-ularly attractive in healthcare, where preferencesmay be both vague and difficult to articulate. The procedures involved, whereby respondents are encouraged to re-think their answers, moveoptions around at various stages and initially define preferences in qualitative terms, isattractive. However, it should be noted that thetechnique relies on computer-based interviews, and will therefore be relatively expensive to carry out. Bryson and colleagues28 argue that QDP is “simple and intuitively appealing”. It has its theoretical basis decision theory, fuzzy set theory, the theory of vague real numbers and voting theory. Given the mapping of quali-tative responses onto an interval scale, it may be argued to posses strength of preferenceproperties. Ngwenyama and Bryson30 note thatgiven the computer-based collection of data,inconsistent answers are not permitted.

CA ranking exercisesIn terms of theoretical basis, the CA rankingapproach was developed in mathematicalpsychology from conjoint measurement theory.32

This theory argues that one of the great strengthsof CA ranking exercises is that cardinal data can be obtained from ordinal rankings.32,398 As Greenand colleagues398 note:

“Recent developments in mathematical psychology,culminating in a new approach to measurements –called conjoint measurement – allow the researcherto measure the separate or parts-worth of each benefit to overall profile value. These approaches use various algorithms to transform the dependentvariable, the preference ordering, into a cardinal(interval) scale and, in the process, develop cardinalscales for the contributory benefits that reflect their relative importance.”

In addition, CA ranking exercises are rooted inLancaster’s theory of value.31

The ranking conjoint approach has undergonerelatively little empirical work in healthcare com-pared with the choice-based approach. Responserates have been good with 59% reported in a


27

postal survey,45 81% in a survey of physicians(including a $50 incentive),37 and 84% and 90% in interviews.42,46 One study reports thatrespondents had little difficulty in completing the exercise,46 two studies reported respondentsranking 18 profiles,37,43 whilst respondents inanother study took only 12–18 minutes to rank 25 profiles.42 Whereas one study reportedbetween 20 minutes to 1 hour to rank 12 profiles,39

another limited profiles to a “manageable” ten.44

No comparisons have been carried out betweenthe CA methods, but one study argues that ranking is easier to complete than choice-based CA (see below for a description of this tech-nique).39 Generally, high levels of consistency have been reported,21,41,43 but further research is needed. In the only study identified that assessed reliability, test–retest coefficients of 0.7and 0.66 were estimated for ten respondents after a 7-month interval between questionnaires.46

Evidence supporting internal validity has beenfound in terms of coefficients on criteria behavingin line with a priori expectations.21,43,44,46 Further,Singh and colleagues46 confirmed their fourhypotheses concerning the influence on differentgroups (the risk conscious group were mostinfluenced by side-effects, the child-focused group were most influenced by the child’s attitude, the cost-conscious group were mostinfluenced by out-of-pocket costs, and the ease-of-use group were most influenced by route oftreatment) and the presence of lower income and education in the cost-conscious group.

Rating techniques

Approaches to eliciting preferencesRating scales within the QALY literatureSupporters of the technique have argued that itsattraction lies in the fact that it has a theoreticalbasis (in psychometrics) and is relatively easy tocomplete (compared with SG and TTO; seebelow).248,399–402 There is, however, debate about the extent to which the VAS incorporates ameasure of strength of preference.246,399,403–405

It has been argued that given that respondents are instructed to place health states on a line such that the intervals between the points reflect the differences they perceive between the health states, the output from this techniquecan be regarded as cardinal.352 However, otherschallenge this view.246,399,403,404 For example, Nordstates that “subjects are clearly ranking states when placing them on the scale, but they express little depth of intention beyond thisordinal level of measurement”.399

Brazier and colleagues253 found the VAS to be the most acceptable of the health-state valuationtechniques, eliciting high response and completionrates.352,359,400,406–411 These findings were supportedby the additional literature identified in our review.Revicki412 and Badia and colleagues365 reportedcompletion rates of 98.7% and 98%, respectively,whilst Morss and colleagues413 reported a 100%completion rate.

Brazier and colleagues253 also noted that the VAS method is less expensive to carry out (since it can be conducted via questionnaire-based surveys) and quicker to complete, but notnecessarily easier to complete, than other health-state utility measures.400,414,415 Some support wasfound for this conclusion in our review. Shiell and colleagues416 observed that many patientsfound it difficult to rate death on the VAS. The authors’ explanation for this was that thepatients completed the exercise with referenceonly to the best imaginable health-state (full health).

The VAS has tested for internal consistency byincluding health states that should logically bepreferred to others, and then testing whether such states are given a higher utility weight. Brazier and colleagues conclude that the VASperforms well according to this definition ofconsistency.253 Similar results were found in theadditional studies we identified. For example,Krabbe and colleagues417 tested the internalconsistency of five health-state valuation tech-niques using the above method. The averageinconsistency for the VAS method was lower(2.0%) than all the other methods (closest wereSG and TTO at 4.6% and 4.3%, respectively). Also, Badia and colleagues365 show encouraginginternal consistency results, citing a significantlylower percentage of logical inconsistencies than the TTO method.

Test–retest results are good for theVAS,359,400,409,418–421 with the only exception beingRutten-van Molken.422 These results are a hybrid of the reviews by Froberg and Kane352 and Brazierand colleagues253 since no additional literature was identified in our review.

With respect to convergent validity, VAS utilityweights have been found to be consistently lower than those estimated using either SG orTTO.400,412,413,422–424 Other evidence concerning the validity of the VAS is highlighted in the Brazier and colleagues report.253 They noted that VAS methods are susceptible to response


28

spreading425 whereby “the respondent seeks toplace (spread) responses across the whole (or a specific) of the available scale”. This is also cited in Kaplan and colleagues407 and Revicki.412

Brazier and colleagues253 suggest that such findings indicate the lack of an interval scalewithin the VAS. The validity of the VAS has alsobeen challenged on the basis that preferences are elicited under certainty, and therefore it isvalue that is being elicited, not utility. However,Dyer and Sarin426 suggest a function that provides a link between value and utility.

Rating scales within CA studiesThe CA rating approach has its theoretical basis in information integration theory and judgementanalysis427,428 where again it has been argued thatcardinal data can be obtained from individualresponses to rating data. In addition, CA ratingexercises are rooted in Lancaster’s theory of value.31

Postal response rates have ranged from 42% to67%52–54,57,61–63,66 and 33% was reported in oneinterview situation.58 Completion rates have beenhigh (78–85%).57–59,66 Harwood and colleagues58

noted that their questionnaires “were probablyabout as difficult as it is reasonable to undertake”,although their response rate was comparable withother scaling methods. Graf and colleagues56

observed that the method could result in fatigue.They noted that rating 16–24 profiles takes around 10–20 minutes and that telephoneinterviews using CA are difficult and the tech-nique is best suited for face-to-face interviews.However, they note that mail surveys are possible.A high level of consistency has been shown, but in limited studies.61,63,64 Similarly, good reliabilityhas been found in two studies but with lownumbers.58,64 Some evidence has been provided to support internal validity61,63 and no evidence of ordering effects has been detected.61

Likert scaleThere is no constrained choice and debate exists as to whether strength of preference ismeasured. Given that there is no assumption ofequal intervals on the scale, the difference between‘agree’ and ‘strongly agree’ may be perceived bythe respondent to be greater than that between‘agree’ and ‘undecided’.9 Likert scales are rela-tively easy to complete, with response rates of84–91% reported in interview or telephonesurveys76,81 and a wider range of 49–96% in postal studies.73,74,77,82 Completion rates of 95% and 70% were noted by Pfennings and colleagues83

and Marks and colleagues,80 respectively. Marks

and colleagues80 further stated that their 20-itemquestionnaire could be completed in less than 5 minutes. Likert scales are often used to makecomparisons between individual statements andcompared across the different respondents and the different issues; however, some researchers gofurther and total up scores across different scales.

Methodological evaluations of Likert scales havebeen limited in healthcare, outside of their use in satisfaction studies. Marks and colleagues80

found good internal consistency and reliability.Construct validity was also supported80 and, in terms of convergent validity, Likert scalesdemonstrated (often significantly) higher resultsthan VAS.83 A major disadvantage of totalling upscores is that “while a set of respondents will always add up to the same score, the same totalmay arise from many different combinations ofresponses, which lead to a loss of informationabout the components of the scale score”.9

Semantic differential techniqueIn a short review of the SDT, Bowles89 suggests that the SDT is acceptable because it requires ashort amount of time to complete and can be used with a variety of populations. This conclusionis supported by the work of Niedz.140 However,Appels and colleagues429 reported a large numberof incomplete responses in two large cohort studies (59% in one cohort and 32% in the other).Like Likert scales, analysis is straightforward,requiring only comparisons of mean and standarddeviation calculations. Mixed results have beenfound regarding the reliability and validity of thetechnique.85,89,92,140,430 Girón and Gomez-Beneyto93

challenge the techniques’ validity. In a study of the relationship between family attitudes andrelapse in schizophrenia, they argue that it ispossible that the “semantic differential does not measure the relative’s attitude but theiraccuracy in assessing the severity of the illness and behavioural disturbance of patients who aremore likely to relapse”. Swain and McNamara92

note that the principle disadvantage of the SDT is the ease with which investigators can chooseadjectives that are inappropriate to the conceptdomain of interest, despite the existence ofmethods to aid appropriate choice. Holmes85

found that SDT suffered from two more generalquestionnaire effects. First, he noted a throughquestionnaire bias, whereby attitudes are modified somewhat from initially more extremepositions; secondly, a within page bias, wherebyrespondents prefer the left-hand side of the page. These limitations may be present with other techniques.


29

Guttman scalesThe Guttman scale derives from psychology, and is rooted in ‘facet theory’.98 It neither incorporatesstrength of preference nor a constrained choice.Because of the lack of published studies, little can be gleaned on the methodological status ofthis method. Response rates of 71% and 65% have been reported, which are good for postalquestionnaires,100,102 although poorer responserates have been reported by psychologists.95

Internal consistency is nearly always reportedbecause of the selection of items for the Guttmanscale. The reported internal consistency is no more than moderate.99,101,102 In addition, it seemsto be common practice to report a ‘scalability co-efficient’, which accounts for the artificial effectresulting from selecting the items for the scale.Again, this was reported as moderate, implying that the published studies were not very good atpicking the right items for the Guttman scale.Reproducibility was not reported and Petersen102

notes that a valid scale was not obtained in his study.

Instruments for eliciting preferencesSatisfaction surveys†, ‡

Satisfaction surveys appear to lack a clearconceptual or theoretical basis, resulting in some confusion over what it is they aremeasuring.143,145,384,431–435 The ‘common sensemodel’, which assumes that patients derive their level of satisfaction by comparing theirexperiences with their expectations, derives frommultiple discrepancy theory.145 However, thistheory has been challenged within the context of satisfaction surveys.111,145 As Carr-Hill notes:“Measured achievements-aspiration gaps … maywell be rationalisations rather than causes ofsatisfaction ratings.”145

Constrained choice is not normally an aspect of satisfaction studies. A further limitation ofsatisfaction surveys is that they provide noguidance on where scarce resources should beconcentrated.143,145 This is a result of such surveysfailing to incorporate any notion of constrainedchoice or strength of preference. For example,suppose respondents were asked to rate theirsatisfaction with five aspects of care (waiting time

for appointment, time in waiting room, time inconsultation, information received and location of appointment) on a rating scale from 0 to 7,where 0 represents completely dissatisfied and 7completely satisfied. Suppose the results indicatedthat patients were most satisfied with time inconsultation, followed by time to appointment and least satisfied with time in waiting room. What then are the policy implications? Shouldresources be devoted to time in the waiting roomsince patients are least satisfied with this aspect of care? The problem with this decision-makingrule is that the characteristic of care that patientsare least satisfied with might also be one that they are least concerned with. Further, despite the fact that patients were most satisfied with timein consultation, they may still prefer marginalimprovements in this characteristic of care overothers that they are less satisfied with. To makedecisions regarding the optimal allocation ofresources, information is required on the weightsattached to the various dimensions that make up ‘satisfaction’.

Despite these properties, satisfaction surveys areclearly a popular method for eliciting preferences.This popularity may partly reflect the fact thatresearchers find such studies relatively easy toconduct, respondents find them relatively easy to answer, and analysts find them relatively easy to analyse. Response rates for satisfaction surveyshave ranged from 38% to 91% with postalquestionnaires.73,103,109,111,113,116,123,131,142,154,156

Whilst it is generally agreed that satisfaction studies provide a relatively low cost researchmethod, Bisset and Chesson suggest that they “may not represent value for money”159 andMcKinley and colleagues highlighted that a well-designed and piloted satisfaction question-naire demands “time and expertise”436 which could make a survey expensive.

Despite their acceptability to researchers andrespondents, a number of problems have beenidentified in the literature. Satisfaction studiestarget patients (service users) and their family/carers, but not the general public. The counter-argument is, of course, that those people with

† The NHS R&D HTA programme has commissioned a separate systematic review on patient satisfaction. This review is entitled ‘The measurement of patient satisfaction: implications for health service delivery through asystematic review of the conceptual, methodological and empirical literature’ (project number 96/27/02) by Crowand colleagues. This report will provide more detailed information of the methodological literature.‡ Satisfaction surveys may be seen as both an approach to, and instrument for, eliciting preferences, depending onwhether the researcher devises his or her own instruments or uses an off-the-peg questionnaire.


30

experience of a particular (health) service are inthe best position to comment up its quality andquantity. In addition, satisfaction questionnairesare said to “encourage patients to respond to his or her own health care on an individual basiswithout reference to the wider collective ofhealthcare users”.431 As with all questionnaire based surveys, the wording and presentation ofquestions in a patient satisfaction survey mayinfluence the responses.437

Perhaps the greatest challenge to the validity ofsatisfaction surveys is the finding that patientsgenerally report high levels of satisfaction with the healthcare they receive.111,145,438,439 Fitzpatrick144

notes that high levels of satisfaction are typicallyrecorded by at least 80% of respondents to satis-faction surveys. Such high satisfaction casts doubt on the ability of such surveys to detect real differences in patients’ opinions.432 Possiblereasons for this include the reluctance of patientsto express negative views about their healthcare440

– also referred to as ‘gratitude bias’ – and a general feeling that ‘what is, must be best’.439

Cleary and McNeill concluded in their review of the patient satisfaction literature that “...frequently only global measures of satisfaction are used”.441 However, OE and smaller scalesatisfaction studies are more likely to report areas of criticism than large-scale (postal)questionnaire surveys.437

Avis argues that despite all the criticism ofsatisfaction studies, “there is a central place forsatisfaction surveys in monitoring standards ofquality in health care. However, they must besensitively performed.”431 Others have highlightedthe need to use research tools that have beenshown to be reliable and valid.437 There are nowseveral ‘off-the-peg’ satisfaction questionnaires inuse.431 Researchers using one of these instrumentscan be assured that the instrument has a certainlevel of validity and reproducibility.

Schedule for the evaluation of individual quality of lifeOnly a handful of SEIQoL studies were identified,and most of those have been for small groups.Thus, little can be concluded in terms of itsmethodological status. Browne and colleagues69

found a response rate of 90% in an elderly group suffering from dementia (n = 56; mean age 74 years), and concluded that these respon-dents were able to understand and complete thequestionnaire. However, Coen and colleagues67

found that only 6/20 patients were able to do so. Some evidence has been found for internal

consistency,67,69,70 reproducibility,68 and constructvalidity67,68 and internal validity.69 However, once again these are with small numbers of respondents.

SERVQUALAlthough SERQUAL has received much empirical attention in the healthcareliterature,166–182 little methodological work has been carried out.370,442–445 The original method employed 22 statements divided into five dimensions on a seven-point Likert scale. It has been observed that, although the tech-nique is “reportedly business neutral”, itemstatements require minor changes in wording to make them appropriate in the healthcaresetting.179,180 Further, researchers have replaced the seven-point scale with a five-point Likert scale, arguing that this leads to less frustration on behalf of respondents, and encourages higher response rates and improved quality ofresponses.177,370 These authors also removednegatively worded statements which reportedly led to confusion and irritation of respondents in favour of positively worded statements only.Others have used a nine-point scale.169,171,446

Later versions of SERVQUAL have incorporated a budget pie question (see below for description of this technique) in which respondents divide 100 points between the five dimensions167,172,447

whilst others have used a 100-point rating scale for each item.171 Hart447 states that the seven-point Likert scale may not have equal intervals and respondents may

“actually deploy an ‘increasing resistance’ model inwhich it is easier (in psychometric terms) to move from the central point (point 4) to its immediateneighbour (point 5) than it is to move from aposition of near perfect satisfaction (point 6) to perfect satisfaction (point 7)”.

Instead, the author suggests assigning cardinalvalues to each point as follows: point on scale 1 (value –6); 2 (–3); 3 (–1); 4 (0); 5 (1); 6 (3); 7 (6).

Response rates have been variable, ranging from 22% to 72% for postal question-naires,170,171,174,179,181,370,442 to 73% by telephone176

and 36–80% when distributed by hand.171,175,180

Raspollini and colleagues181 observed that whilst it is not necessary to train interviewers,interpretation and analysis of results are “difficult” and “complex”.

Evidence of internal consistency and validity were supported in the marketing literature,165

and Dyck448 argues that these are preserved if


31

“the intent and order of the questions remains thesame”. There has generally been good evidence of the internal consistency of the dimensions ofSERVQUAL.167,168,173,176–178,180,370 Some studies sup-port its validity and applicability to healthcare,167,173

whilst others dispute this, challenging its constructvalidity and claiming intercorrelation of its dimen-sions.179,443 One study by Oswald and colleagues442

found that dimensions of quality assessmentdiffered from those assessed in SERVQUAL. Duffy and Ketchand182 found a large unexplainedvariation in service satisfaction, concluding thatdimensions that may have influenced satisfactionwere excluded.

Choice-based techniques

Approaches to eliciting preferencesSimple choice exercisesIn terms of acceptability, Charny and colleagues183

reported that whilst 31% of respondents had made all 13 choices, nearly 69% were unable to complete all the choices. However, this find-ing may relate to the sensitive nature of thequestion being asked. Internal validity was alsofound in the related studies as respondentsfavoured the young over the old, women over men, non-smokers and non-drinkers over smokers and drinkers.183,184 In the study by Charny and colleagues,183 the instance where an 8-year-old child was favoured over a 2-year-old was explained by the greater parentalinvestment in terms of effort and emotion.

Regarding RPS, Ryynanen and colleagues186

observed that members of the public completedthe questionnaires quickly. However, most haddifficulty in making choices between two altern-atives, some found that the questionnaire madethem anxious and one reacted aggressively. How-ever, these findings may reflect the sensitive natureof the questions being asked. Medical and nursingundergraduates completed the questions quicklyand without difficulty. No information is availableon internal consistency. Eight respondents answer-ing three different sets of RPS questions measuredtest–retest reliability. The authors concluded that reliability was “good”.186 Ryynanen andcolleagues186 found comparable results in terms of criterion validity between conventional and RPS questionnaires in an undergraduate student population.

CA choice-based questionsThe increased number of applications of choice-based CA has been accompanied by an

investigation of many of the main methodologicalissues that are important when developing atechnique in a new setting. Choice-based CAexercises are held to be acceptable to individualson the basis that they present them with the types of decisions they face on a daily basis. It is this argument that has led to the choice-based technique being preferred over ranking and rating approaches.52,60,61,194,206,449 Choice-based techniques have also been favoured byeconomists because of their underpinnings in a branch of economic theory known as randomutility theory.450,451 However, it has been noted in the literature that the number of choicespresented to individuals should not exceed 12 (at maximum), and the number of criteria should not exceed five or six. More choices, or criteria within the choices, will result indifficulties in completing the questionnaires.

Choice-based CA studies have mainly been carried out using postal questionnaires, pro-ducing response rates ranging from 18% to 88%.Response rates are generally higher if question-naires are sent to individuals involved in a ran-domised trial, include financial incentives, areaccompanied by a physician’s covering letter, orare given out in clinics.188,191,193,196,197,199–203 Fewdifficulties have been reported when answeringchoice-based CA questions191,194,199,201–203,396 and two studies report that the technique was wellreceived by policy-makers.198,205

Many choice-based conjoint studies have includedtests of internal consistency by including dominantoptions, i.e. options where one scenario is con-sidered ‘better’ than the one it is being comparedwith on all criteria. Respondents would thereforebe expected to choose this scenario. Tests of thisnature suggest a low proportion of inconsistentresponses.193,194,196,197,199,202,203,205–207,396 One study was identified which tested internal consistencywith reference to transitivity, i.e. if A is preferred to B and B is preferred to C then A should be preferred to C.193 In this study, 6% ofrespondents failed at least one of the consistency tests.

Only one study was identified in the healthcareliterature which had assessed the test–retest reli-ability of choice-based CA. In a study looking atparent preferences for out of hours care, SanMiguel and colleagues357 found a high level ofreproducibility within a 2-month time period.

The validity of choice-based CA has beenaddressed at a number of levels. The method


32

has demonstrated high levels of internalvalidity.187,190,194–197,199–203,207 Convergent validity has been with SG and VAS,192 and WTP.194,199 Anumber of studies have tested for framing effects.Whilst Vick and Scott203 found some evidence of an ordering effect with regard to the criteria,202

two studies have found no evidence of framingeffects.61,189 Ryan and Wordsworth200 found WTPestimates derived from choice-based CA to beinsensitive to the level of criteria.

CA assumes that individuals have continuouspreferences such that there is always someimprovement in one attribute that can com-pensate an individual for a deterioration in the level of another attribute. Tests are oftencarried out to see if this assumption holds.193,195,199

There is less agreement in the literature con-cerning how to deal with individuals exhibit-ing non-compensatory behaviour. Some studiesdrop such respondents, arguing that marginal rates of substitution (the rate at which they tradebetween criteria) cannot be estimated for them(since they are not willing to trade) and thereforethe utility function estimated is inappropriate.194

Others compare estimated utility functions with and without non-traders.187

Analytic hierarchy processThe AHP has been shown to have an axiomaticfoundation and to produce output on a ratioscale.218,219,223,225 The technique typically collectsdata using interactive techniques (often computer-based).452 Whilst this may increase the cost of datacollection, it can also ensure the consistency ofresponses (see below).

Studies employing the technique, mainly in a face-to-face type of interview, suggest a high level ofacceptability. Schwartz and Oren225 reported that83% of patients completed and understood aquestionnaire that consisted of 28 pairwise com-parisons, and Peralta-Carcelen and colleagues452

reported that 90% (83/92) of pregnant women,51% (40/78) of obstetricians and 67% (40/60) ofpaediatricians participated in their study. Withinthis study, 89% of the women and 90% of thephysicians were reported to have understood the interview format, although there were somedifficulties reported. However, these appear toconcern the nature of the data presented to themand not the method. The authors further note the limited educational background of theirsample of women but noted they did not haveundue difficulty in understanding the model andthus claim that this justifies the generalisability of the method. Dolan235 directly addressed the

question of whether patients were capable andwilling to use AHP in medical decision-making.Following a structured interview, respondents ere followed up with five evaluative questionsconcerning the technique. Despite the smallnumbers (n = 20), positive findings were found. In all, 90% (18/20) of respondents were defined as “capable and willing” to complete the exercise,and Dolan concluded that AHP was “likely to beacceptable to and within the capabilities of manypatients”. Positive findings have been foundelsewhere, albeit with small numbers, as havefurther cited studies (published in abstract form only).232,453–455

Javalgi and colleagues239 conducted a postalquestionnaire to assess factors important inchoosing a hospital. They justify a mail survey on the basis of cost and the nature of the datarequired. A 47% response rate was reported(235/500) with 220 responses being usable.

The introduction of the software has facilitated the application of the AHP in terms of thecollection and analysis of data (with weights being automatically estimated).226,235,237,239 Thiscomputer software also has in-built tests ofconsistency. Consistency in the AHP refers to theaxiom of transitivity, i.e. if A is twice as preferableas B, and B is three times more preferable than C, then, to be consistent, A must be six times morepreferable than C. A consistency ratio is produced,with a value of 0.1 or less being defined as accept-able.227,240,452 High levels of consistency havegenerally been reported. Peralta-Carcelen andcolleagues452 found that of 83 pregnant women, 40 obstetricians and 40 paediatricians, only threeof the women were inconsistent. Schwartz andOren225 excluded 5% of respondents from theanalysis since their consistency ratios exceeded0.50 (the remaining ranged from 0.11 to 0.23).Finally, Hannan and colleagues241 argue that their responses could not have been randomly allocated.

Very little work has addressed issues related to the validity of the technique. Schwartz andOren,225 in a study looking at patient preferencesfor treatment of myocardial infarction, note thatthe choice model accurately predicted behaviour.Dolan235 notes that the greatest challenge to thevalidity of the technique comes through the rankedreversal phenomenon. Here a change in therelative desirability of a criterion/alternative iscaused by the introduction of another criterion/alternative. However, a new method of AHP hasbeen developed which precludes rank reversal.229,235


33

Dolan226 notes that this method has successfullybeen used to promote a change in behaviour in patient care§.

Standard gambleThe SG technique has often been argued to be the gold standard by economists since it is rootedin expected utility theory (EUT).49,245,352,456,457 Fromthis theory, which explains individual decision-making under uncertainty, it is then held that theutility weights derived from the exercise representa strength of preference measure.458 However,given the known problems individuals have withprobabilities, and the nature of the questionsposed (with the individual required to specify aprobability level of indifference), SG exercisesmust be carried out via an interview, with props,and the interviewer must be trained.253 Given this, the technique is likely to be relativelyexpensive.253,352,400

The review by Brazier and colleagues253 found high levels of response rates and completion rates,ranging between 80% and 100%.253,361,411–413,459–462

Similar conclusions were derived from the addi-tional literature we identified in our review.365,458

Shackley and Cairns458 reported a 95% responserate and Badia and colleagues365 a 98% rate. Itmust be remembered that such rates are forinterviews where completion rates would beexpected to be relatively high.

Brazier and colleague253 note that where com-pletion problems are reported when using the SG method, these difficulties are no worse thanthose faced by other techniques. They highlightthe problems encountered by Patrick and col-leagues463 and van der Donk414 where it was statedthat the SG was no more burdensome than eitherthe TTO or VAS methods. However, in a studyidentified in our review, Hall and colleagues,464

in comparing VAS, TTO and SG, found SG to be the most difficult technique to understand.Richardson403 observed that “empirical evidencesuggests that people have difficulty understandingthe objective meaning of [SG] probabilities,especially extreme values”. This is a view shared byJohannesson and colleagues404 and Froberg andKane,352 who both state that some decisiontheorists see the SG as being too difficult toexplain to respondents.

Tests of internal consistency have been carried out in the same way as for the VAS, and the results are mixed.253,352,361,465 Dolan andcolleagues362 in testing SG against 12 logicallyconsistent comparisons, argued that the techniqueproduced high levels of consistency. This studyincorporated a test between two variants, oneinvolving a prop to aid the decision-maker (aspecially designed board and cards) and thesecond involving a self-completion booklet with no props. Consistency rates for SG with props was 83.8% and 87.5% with no props. Llewellyn-Thomas and colleagues361 also found a high levelof consistency, with 84% of respondents (54/64)ranking five health states in accordance with a priori expectations. However, Froberg and Kane352

argue that given that people find it difficult towork with probabilities, and the fact that they may also have an aversion to taking risks, responses are often inconsistent. This conclusion was supported by Thompson.465

Brazier and colleagues253 reported good test–retestresults for the SG technique.362,400,409,418,419,422,459,466

No additional studies were identified in our review.

Mixed evidence exists concerning the technique’svalidity. In terms of convergent validity, Brazier and colleagues253 reported that whilst SG has been shown to correlate reasonably well withTTO,362,466–469 it does not correlate well with either health status or the VAS.409,422,456 O’Brien and Viramontes419 found that in comparing SG to VAS and WTP, SG was most highly related to the latter. An additional study we identified in our review, by Bala and colleagues,470 found SG to be consistent with WTP.

SG is based on an assumption that individuals arewilling to trade risk. Individuals who exhibit non-trading behaviour may do so either because theyare completely risk averse, always preferring thecertain intermediate outcome to the gamble, orbecause they are risk lovers, always preferring agamble to the certain health outcome. Suchbehaviour would provide information concerningthe value of the health state being considered.However, if non-trading is a consequence of therespondent objecting to taking risks, the valuesestimated from a SG experiment will not indicateutilities of given health states.471

§ The question of whether to preclude rank reversal depends on the reason for the rank reversal. If rank reversaloccurs because respondents do not understand the technique, this may be a reason to exclude such cases. However, if rank reversal occurs because of a change in preferences, then such exclusion is not valid. This issue needs moreinvestigation (alongside the increased use of AHP in healthcare).


34

Related to the issue of trading risk, a concern thathas been expressed in the literature concerningthe SG technique is that, when asking individualsto trade, the attribute that is being traded becomessalient. That is, people give more importance tothis attribute.213 This has been shown within SGexperiments in which individuals are first askedtheir probability indifference point between agamble and certain outcome and are then pre-sented with a pairwise comparison between thesetwo scenarios (i.e. the certainty equivalent and the gamble to which individuals said they wereindifferent). In such choice experiments thecertainty equivalent is usually preferred to thegamble.472,473 One possible explanation for this is that the attribute that is traded (risk in thisexample) is weighted more highly than in asituation where a choice is made.

Questions are therefore raised concerning whetherindividuals who are not willing to trade are pro-testing to the technique, are giving an exaggeratedimportance to the attribute being traded, or reallydo not value the health state being valued. This has not been addressed in the literature.

Perhaps the biggest challenge to the validity of the SG technique is the wealth of evidenceshowing that individuals consistently violate the axioms of EUT.352 Brazier and colleagues253

provide a review of this literature. This raises a question concerning the justification for thecontinued use of SG. The answer to this questionmay lie in the normative appeal of the technique.That is, violations of the axioms of EUT may bedue to misunderstandings on the part of thosemaking choices, and individuals may be willing to receive help from the theory to make optimaldecisions. However, there have been increasingcriticisms of EUT as a normative model ofdecision-making under uncertainty.474–481 This is fuelled by the fact that, when asked to reviseresponses that violate the independence andreducibility axioms, many individuals refuse.478

(Experimental evidence suggests that individualswill change preferences when alerted to intransi-tivities.474) Alternative theories of decision-makingunder uncertainty have thus been put forward.These argue that EUT fails not because individualbehaviour is suboptimal but because the theoryfails to take account of psychological factors ofregret and disappointment.476,477,479–481 Evidence has suggested that regret is an important elementin individual valuation and decision-making inhealthcare.482,483 Questioning of the normativemerit of EUT means questioning the utilityestimates derived using the SG technique.

Time trade-offAs with the SG, the TTO method inherentlyprovides the respondent with a constrained choice that elicits utility numbers that are held torepresent strength of preference. The TTO tech-nique was developed by Torrance and colleagues247

specifically for use in healthcare research as asimple-to-administer alternative to the SG,352

and a “less complicated, conceptually differentalthough equally sound alternative to SG”.417

Like SG, the technique requires interview-basedsurveys so it will be relatively expensive. Whilst the technique has no well-defined theoretical basis, it does involve the concept of choice, andtherefore may be argued to be rooted in welfare economics.

Brazier and colleagues253 report good rates ofacceptability.363,462,463,484–489 For example, Detsky and colleagues487 report a 97% completion rate,Johnson and colleagues485 100%, Dolan andcolleagues362 95% and Glasziou and colleagues488

91%. Similar results were derived from the twoadditional studies identified in our review: Zug and colleagues469 reported a completion rate of 94% and Handler and colleagues490 100%.However, as with the SG, it should be noted that such rates are for interviews so they would be expected to be high.

There is a limited amount of literature on theinternal consistency of TTO responses. Brazier and colleagues253 report Dolan and colleagues362 ashaving a 92% consistency rate against 12 logicallyconsistent comparisons. In addition, our reviewidentified a study by Badia and colleagues365 thatconcluded that the TTO technique produced thehighest level of inconsistencies when compared to the VAS and a ranking method.

For the reproducibility criteria, test–retests aregood for the TTO.362,363,400,418,420,459,462,466,491,492

Brazier and colleagues253 note that in comparingthe test–retest measures for each of the health-status measures, the TTO probably presented thehighest reliability.

In terms of convergent validity, Brazier andcolleagues253 highlighted reasonable correlationbetween TTO and SG valuations.466–469,493 Brazierand colleagues253 also cited Robinson andcolleagues494 where comparisons were madebetween the TTO and VAS. It was found that TTO valuations were a better reflection of health-state preferences than were VAS scores.Additional literature identified in our review by Zethraeus and colleagues495 and Swan and


35

colleagues496 both found that the TTO scoresconverged more with WTP than did rating scales.

One of the challenges to the validity of the TTO technique is in the techniques underlyingassumption of constant proportionality. That is, the technique implicitly assumes that the amountof time an individual is willing to give up to be in a given health state is independent of the timehorizon in which they will be in that health state.So, if an individual states that they would give up 2 years of a 10-year time horizon to be in agiven better health state, then the assumption of proportionality implies that they would give up 4 years from a 20-year time horizon. How-ever, it is possible that the value an individual gives to a certain health state is influenced by the amount of time they have to spend in that state.49,462,497–500

A further underlying assumption of the TTOtechnique is that individuals are willing to tradelife expectancy. However, Brazier and colleagues253

identified literature that indicated that individualswere unwilling to do this.484,490,494,501 For example,Irvine501 revealed that 47% of respondents did notaccept a reduction in life span in order to obtain a shorter period in optimal health and Robinsonand colleagues494 reported that the TTO healthstates have a “threshold tolerability” whereby acertain period of time had to be exceeded beforerespondents would be willing to trade. As with theidentification of non-trading in SG experiments,this may be a manifestation of the potentialsalience problem. As with SG, questions aretherefore raised concerning whether individualswho are not willing to trade are protesting to the technique, are giving an exaggerated import-ance to the attribute being traded, or really do not value the health state being valued. As with SG, this has not been addressed in the TTO literature.

An additional study identified in our review founddifficulties when applying the TTO technique toassess patient preferences for chemotherapy.416

However, satisfactory results were obtained whenthe same analysis was performed with patientsreceiving hospital dialysis for end-stage renalfailure. The authors note that “this is typical ofTTO exercises as the approach is usually advocatedfor eliciting preference weightings for chronichealth states … and while such patients may live

with the cancer, or fear of its recurrence, for someconsiderable time, it is unlikely that any will bemaintained on long term chemotherapy”. Thisview is highlighted by Dolan and Gudex.502 Theystate that the “TTO method is only feasible forvaluing chronic health states that last for durationsof five years or more”.

Another concern for validity is found in Dolan and Gudex.502 Here, evidence of framing effectswere cited, arguing that values elicited using thistechnique may be a function of the questionnairerather than the respondents’ underlying prefer-ences. Concern has also been expressed that theTTO presents individuals with an unrealisticchoice, since the choice is framed in a certainty context.

Person trade-off #

Given the PTO’s consideration of distributiveissues, it is intuitively appealing.251 However, it has been criticised for a lack of any theoreticalbasis.49,250,503 Given that the choice question isphrased in a societal context, it cannot be seen to be rooted in economic welfare theory (a theorythat has been linked to the TTO technique).However, in support of the technique, it has been argued to possess interval properties.250

Whilst very little empirical work has been carried out on the PTO technique, Brazier andcolleagues253 argue that the technique is promising,and future research should be carried out. Theevidence presented by Brazier and colleaguesindicates that if the PTO technique is to besuccessfully used within healthcare, it should bewithin an interview format.248,253 For example, Nord and colleagues504 report response rates of 28.2% and 27% for self-administered question-naires and Nord251 notes that PTO responses from17 of the 53 respondents showed an unwillingnessto take part in the PTO task. He also notes that the technique can be cognitively demanding,subject to framing effects, and inappropriate foruse with self-administered questionnaires. Ubeland colleagues364 also report difficulties using thePTO via a written survey, reporting high levels of inconsistency.

Brazier and colleagues253 note that there iscurrently a lack of information on the reliability of the technique, and very limited information on validity. Patrick and colleagues248 found

# Given that our review identified no additional studies, this section presents a summary of two reviews, that by Brazierand Deverill49 and an update of this by Green.250


36

some evidence of convergent validity with respectto PTO and VAS, and Froberg and Kane352

comment on a study that found the VAS and PTO to give similar results.

Willingness to payWTP has been widely used in healthcare to elicitpublic views.256–258 The application of the WTPtechnique to healthcare has focused on using thePC and CE approaches.259–263

WTP has its theoretical basis in welfare economictheory.505,506 Pauly506 observed:

“In welfare economic theory, there is only oneaccepted way to measure the benefits an individualgets from a program. Benefit is assessed as thewillingness to pay for the programme when suppliedwith information as complete as it can be.”

Given that benefits are measured in money, it isheld to represent a cardinal measure.

The different approaches to WTP have revealeddifferent levels of acceptability to respondents. The OE approach has prompted large numbers ofitem non-response and protest answers.257,261,507–511

In comparison, the PC and CE approaches havedemonstrated higher response rates,261,367,506,512,513

lower item non-response260,261,263,514–517 and less zero responses.512 The bidding game has also given high response rates in an interview settingwhere it is best suited because of its complexityrather than a postal questionnaire.353,518 Whilstpostal questionnaires are considerably cheaperthan interviews, they generate lower response rates. Flowers and colleagues519 demonstrated the viability of automated computer-basedquestionnaires. There is some evidence of low response rates in older age groups.470

There is much evidence through directquestioning of respondents and interviewers that respondents have few problems in under-standing the WTP questions.505,507,519–523 On theother hand, there is also some evidence thatrespondents find the WTP questions difficult toanswer,355,524 and concern has been expressed that individuals will object to the technique on the basis of the technique being related to abilityto pay. In the most comprehensive WTP project in healthcare, Donaldson’s354 six country European study found the WTP method to be “feasible”, with only one country having more than 10% protest responses. However,Ryan517 noted providers’ opposition to the WTP technique.

One way of measuring the consistency of WTP is to ask respondents their preference for thecommodities being valued and then compare such preferences with their stated WTP. Arespondent that gives a smaller WTP value for their preferred option or greater WTP value fortheir less preferred option may be consideredinconsistent. Studies using the WTP techniquehave shown inconsistencies between WTP valuesand stated preferences.355,367–369,417,525 For example,Ryan and San Miguel,369 in looking at women’spreferences in the treatment of menorrhagia,found that whilst women stated that they preferredconservative treatment to hysterectomy, they were willing to pay more for the latter. In thisstudy, cost-based responses were found to partlyexplain these inconsistencies. A number of otherWTP studies have found evidence of cost-basedresponses. Schkade and Payne,526 in using verbalprotocol analysis to investigate how people respond to WTP questions, found that individualsjustify their responses by referring to the cost ofthe commodity being valued. Donaldson andcolleagues355 found evidence of cost-basedresponses in a study looking at different methodsof testing for cystic fibrosis. They argued that this problem may arise when OE questions areused to elicit WTP values. However, the study cited by Ryan and San Miguel found evidence of cost-based responses when using the PC.369

A number of studies have found evidence of goodtest–retest reliability,354,419,519,527,528 although othershave found the opposite.417,508 Klose concluded that reproducibility remains an important area for future research.257

Whilst some studies have found strong indicationsof internal validity,257,470,529–531 others have shownmixed results263,355,368,465,507–510,518,532–535 or opposite toexpected results.367,511,536–540 Regarding convergentvalidity, Klose noted evidence of correlation withother measures of health benefits found in somebut not all studies.419,495,520 These include ratingscales,417,495,496,513,541,542 SG,417,419,470,505 TTO,417,495,496,505

and choice-based CA.194,199

Two studies were identified which had tested thecriterion validity of the WTP technique. Granbergand colleagues543 found that 55% of couples gave aWTP of more than £10,000 for in vitro fertilisation(IVF) treatment, which equates to the true cost ofthis non-NHS treatment (it should be noted thatIVF is available on the NHS but access is restricted).Walraven,544 in an empirical study in Tanzania,found a majority of patients actually paid morethan they said they were willing to pay: 62% at one


37

hospital and 67% at another. At another hospital,which later introduced user charges, WTPpredicted behaviour “reasonably well”.

Concern has been expressed concerning thevalidity of the WTP technique for a number of reasons. One of the greatest challenges concerns the evidence of the insensitivity of WTP to the size of the healthcare interventionbeing valued.262,354,356,509,530,533,545–551 However, a fewstudies challenge this.552–554 Whilst no evidence has been found of question order effects,354,553

there is evidence suggesting sensitivity to thepayment vehicle,548 as well as the technique used to estimate WTP.354,555,556

The WTP approach used may determine theextent and type of any biases found. Several studies detected the presence of ‘yea-saying’ withthe CE approach¶,263,356,555 some evidence of rangebias with the PC355,506,557 and starting point bias with the bidding game.353,356,552,558 Further researchhas shown the results of the CE approach to besensitive to methods of analysis, including treat-ment of ‘don’t know’ responses, limits of inte-gration and bid vector design.559,560 Evidence hasalso been found of strategic behaviour.532,544

Measure of valueGiven the obvious lack of use of this technique in healthcare, there is a limited amount ofmethodological work. The technique wouldobviously need to be carried out within an inter-view, so could be potentially costly. It may also be a lengthy and tiresome process if there are manyalternative options to compare. Dickinson265

considered the investment of 15 hours of manage-ment time involved as a “good buy”. However, ithas a number of attractions, not least the attemptto achieve the optimal bundle within the resourcesavailable. To this end, the technique incorporatesthe important notion of scarcity. The methodologyof the technique would ensure internal consist-ency. There is no information currently availableon reliability or validity, and research around these issues should be encouraged.

Allocation of pointsWhilst little work has been carried out looking at the budget pie, the characteristics of the tech-nique suggest that it could be potentially useful for setting priorities. Given its obvious ease ofuse,561 it is relatively inexpensive to carry out(because mailed questionnaires can be used).282

Whilst a theoretical basis for the technique has not been developed, individuals are explicitlyforced to think about trade-offs and strength of preference. Indeed, Clark277 claims that giventhat the technique uses money (or points) togenerate trade-offs, the resulting measure iscardinal. The case for this is strengthened by the fact that respondents are told to think abouttheir strength of preference when allocatingmoney/points. Clark277 reports unpublished work by Ostrom which found high response rates across educational backgrounds, whilstStrauss and Hughes282 reported a response rate of 28% (1001/3517) from postal questionnaires.Test–retest reliability was declared as “modest” by Srivastava and colleagues562 (their question-naire included Likert and ranking questions).Clark277 states that although the budget piemethod is tentatively appealing, results depend on conversion of money to utility, a directrelationship between money and utility, and honest revealing of preferences. Convergentvalidity between the budget pie and Likert andranking questions have been shown to be poor.562

Mullen and Spurgeon563 in their review of thebudget pie technique and its use in the health-care noted that scaling problems could beencountered when using the budget pie literally,i.e. dividing or allocating monetary budgets. If the budget is going to be realistic, the ability ofrespondents to cope with sums that are perhapsbeyond their normal experience is compromised.By the same token, using unrealistic sums mayresult in unrealistic responses. This has a bearingon whether realistic or ‘honest’ preferences can beelicited, which in turn may mean that cardinality is not maintained and intensity of preferences isnot accurately reflected in individuals’ responses. It has therefore been more common in recentstudies564 to define a ‘budget’ in terms of a set of tokens or points that must be divided orallocated across alternatives.

Instruments for eliciting preferencesAllocation of pointsThe PGI has been assessed against the short-form36 (SF-36) for convergent validity269–271 and beenshown to perform reasonably well. It is repro-ducible for group analyses but not for comparisonof individuals as assessed by test–retest.269–271 Itelucidates relevant items on a much wider rangethan in pre-set techniques.272,273 The concern with the PGI is its acceptability. Correctlycompleted response rates are low: of a response

¶ Yea-saying refers to the tendency to say yes to whatever you are asked.


38

of 75.4% in the original study of back pain, onlyhalf completed it correctly,269 and Macduff andRussell270 reported similar findings. Thus, while ithas the advantage of offering individual choice, itis not yet usable in practice, especially as a postaltool for prioritisation.270

The SEIQoL–DW is reported to be more compre-hensible and acceptable than the original SEIQoLor PGI, but it is interviewer-delivered by cliniciansor researchers (both of whom like it274,565), whomay therefore be able to ensure understanding.The two methods are not interchangeable becausethey are conceptually different.565 As noted above,SEIQoL elicits implicit judgements whereas, bydefinition, DW is explicit. However, the authors use the former when there is no time constraintand the participants are not cognitively impaired.


The systematic review identified a number ofquantitative techniques that have been used foreliciting public preferences. Techniques identifiedwere classified as either ranking exercises, ratingexercises or choice-based exercises. An importantdistinction was made between instruments andapproaches for eliciting preferences. Conclusionsregarding the methodological status of techniquesare stronger for the former.

Ranking exercises included simple rankingquestions, QDP and CA. The popularity of simple ranking exercises probably reflects the ease of both devising a ranking exercise andanalysing the resulting data. Whilst it is recognisedthat ranking exercises may provide policy makerswith some useful information, their failure toincorporate any concept of opportunity cost, lack of any well-developed theoretical basis andlack of any measure of strength of preference,renders them of limited practical use when setting priorities.

The QDP has not been used to date in healthcare,and therefore there is very little empirical workassessing the methodological status of the tech-nique. One of the potential strengths of thistechnique is its ability to deal with the vaguenessthat exists in human decision-making. This ispotentially relevant in healthcare, where prefer-ences may be both vague and difficult to articulate.Given its well-developed theoretical basis, and itsstrength of preference properties, future workeliciting public preferences should consider this technique.

CA (ranking) exercises are potentially very usefulat the policy level. From such output it is possibleto estimate the relative weights of different attri-butes/criteria in the overall provision of a good or service, the trade-offs individuals make betweenthese attributes, and an overall benefit or utilityscore for different ways of providing a good orservice. The ranking approach is attractive in thatit has a well-developed theoretical basis, andcardinal data can be obtained from ordinal data.Whilst the ranking approach has received littleattention in healthcare (relative to the choice-based approach), future work should explore the use of this technique.

A number of different rating scales, andapplications, were identified. The VAS has provedpopular as a method for estimating quality weightswithin the QALY paradigm. Whilst the technique is attractive in terms of the relative ease with which quality weights can be estimated, its mainlimitations are the lack of any constrained choiceand the doubts expressed over whether thetechnique measures strength of preference. Given the availability of alternative techniques, the suitability of this technique for estimatingquality weights must be questioned.

CA rating scales have been applied in healthcare,and the limited methodological work availableindicates that the technique did well against thepredefined criteria. As with ranking exercises, one of the attractions of this approach is theelicitation of cardinal measures of benefit. Futurework should explore the possible uses of thistechnique. A variation of CA rating scales is theSEIQoL, a technique used in the health outcomeliterature to estimate quality of life scores forpatients. There is currently a lack of empiricalwork on the methodological status of this instru-ment. However, given it is essentially the same as CA rating exercises, future work should be encouraged.

A number of techniques have been applied bysocial and behavioural scientists to elicit attitudes.Likert scales are most commonly used, but studieswere also found which had employed Guttmanscales and the SDT. Whilst such scales provideuseful information, it must be recognised by usersthat there is no strength of preference measure,nor any consideration to the importance of thedifferent components that made up the score.

Satisfaction surveys have been frequently used toelicit public opinion. Their popularity probablyreflects the relative ease of carrying out such


39

surveys. Researchers should ensure that theyconstruct sensitive techniques, or else use genericinstruments where validity has already beenestablished. Even when well-designed, sensitivetechniques are used, users should be aware of thelimited use of such techniques at the policy level.SERVQUAL appears to be a potentially usefulinstrument and future research should consider its application in healthcare.

A number of choice-based techniques wereidentified. Three such techniques – MoV, AHP and allocation of points technique – have hadlimited application in healthcare, resulting in asmall literature on their methodological status.However, they all appear potentially useful. TheMoV technique, although requiring an interviewsetting, considers the optimal allocation ofresources within a given budget. AHP has a sound theoretical base and incorporates intensityof preference. The apparent simplicity of theallocation of points technique, together with its

explicit attempts to consider trade-offs andstrength of preference, suggest that it should form the focus of further research. Whilst the SG technique has been argued to be the goldstandard as a method for eliciting utility weightswithin the QALY paradigm, concern is expressedover the evidence suggesting that individualsconsistently violate the axioms on which it is based. TTO, discrete choice CA and WTP all did well against the predefined criteria and should continue to be researched. Moremethodological research is required on the PTO.

A number of techniques have been identifiedabove for both applications in healthcare andfuture research. A priority area of research is to address the cognitive strategies and decision-making heuristics respondents adopt when com-pleting quantitative surveys. This should involveextensive qualitative work to inform the design and interpretation of quantitative studies.


41

This review concentrates on qualitative methods that have been, or could be, used

to elicit public opinion regarding priority setting in healthcare. The qualitative techniques identifiedin chapter 3 are reviewed according to the criteriain chapter 4. Techniques were classified as eitherindividual-based or group-based. It was recognisedthat the terminology of the chosen criteria wouldnot have been used in most existing reviews ofqualitative methods, and that therefore therewould be an element of interpretation or trans-lation in our review*. Given the nature of quali-tative research, all methods identified may be seen as approaches to eliciting preferences (as opposed to instruments).

This chapter sets out techniques in sequence. For each technique general issues regarding themethodological status of the technique are firstdiscussed. Secondly, where identified, examples are given of how the technique has been used inprimary research, and methodological issues arehighlighted in respect of the chosen criteria. Inthis second part, papers have been included in thereview only if they adhere to the inclusion criteriaset out in the methods section (chapter 2); that is,if the author/authors conducted a primary pieceof research and made some attempt, eitherexplicitly or implicitly, to evaluate it. Where acomment about a criterion has been madeexplicitly by an author/authors it is included here only if it is consistent with our terminology.Where a comment is made implicitly, that is if it fits in with our criteria but the exact word is not used, it has also been mentioned, but isreferred to using our definition of the term.

Individual approaches

One-to-one interviewsGeneral methodological issuesThe methodological literature reveals that thereare several strengths to the interview method. The interviewee has the opportunity to control

the direction in which the interview is heading so that true feelings can be determined, thusincreasing the potential validity of the technique.381

Because the interviewer is present ambiguities canbe clarified and misconceptions checked at thetime of the interview.9,381 Interviewees are offeredsufficient time to express their ideas and have theopportunity to ask questions and have these fullyanswered.307 They provide those people who arenot particularly vocal in public settings, in a public meeting for instance, with an opportunity to offer their opinions about a certain topic.“These may offer a way of enabling the silent voices to be heard.”1 Ultimately, “the only valid way of hearing the voice of the public may be by one to one interviews”.566

However, weaknesses have also been noted.Respondents may give socially desirable responses,wishing to please the interviewer and expressingideas that they think the interviewer is seeking.567

There is a general problem with interviews in thatthey are not an anonymous research method351

and therefore some people may withhold inform-ation, particularly if the topic is very personal orembarrassing. Respondents might be looking forconfirmation from the interviewer in response to their answers. There is, of course, also risk ofinterviewer bias.371 Objectivity is another problem,given that the researcher can never be trulyobjective; this should be recognised and stepsshould be taken to minimise these factors.9

The method of analysis is important for reliability,validity and objectivity. Interviews may be recordedand transcribed and recurrent themes identified,384

or the interviewer can simply make journalist-typeshorthand notes during the interview. The decisionregarding recording interviews should take theconcerns of the interviewee rather than those ofthe interviewer into consideration, but, if the twodiffer, neither is regarded as more valid, reliable orobjective than the other.384 It is widely believed thatdisplaying statements verbatim helps to give thereader a feel for the kinds of comments made in

Chapter 6

A review of qualitative techniques for eliciting public views

* Given this, the evaluation of qualitative instruments may be more open to problems of inter-observer reliability.


42

the respondent’s own words. There are, however, a number of problems associated with this format. There is a danger that quotes can be taken out of context or become distorted.9 Cost is another consideration regarding conducting and transcribing interviews. It is suggested, forexample, that 4 hours or more be allowed for the transcription of every 1 hour of recordedmaterial568 and the reporting of the findings can take 4–6 weeks.379

Examples of the application of the techniqueInterviews have been very widely used in health-care research. Here several examples and thespecific methodological problems encountered are presented as illustrations. Additional examplesare highlighted that look at using the telephone,email and the dyadic format.

In the study by Ayanian and colleagues,288

patients were selected from a stratified randomsample of patients undergoing dialysis. They were interviewed by telephone about issuessurrounding their treatment and attitudes towardsit. One major drawback of interviewing in thisstudy raised by the authors was that the sampleobtained may not have been sufficiently repre-sentative of racial issues precisely because a portion of the intended sample was debarred from taking part because they could not speakEnglish. By the criteria of this review, this con-cerns the generalisability of the results. Addition-ally, results achieved might have been due to thedoctor who was advising the patients; that is,differences could be attributable to the doctorsbeing of a different ethnicity to that of the patient.This would affect the validity of the study.

The study by Dicker and Armstrong289 involvedsemi-structured interviews with 16 patients from an inner-city general practice. Steps were taken tocompensate for the small number of participantsby ensuring that a genuine cross-section of thecommunity was recruited. Thus, representativenesswas sought by deliberately selecting a range ofdifferent people using the service – a samplingtechnique known as purpose or purposivesampling.8 This study found that respondentsanswered in relation to the context of the choicesand considered a more general approach detachedfrom their own motives. Here the criterion ofacceptability was addressed: the authors reportedthat participants found it unacceptable to considerthe choices from a personal perspective. This wasperceived by the authors as a way for participantsto appear unselfish, or to show a genuine concernfor their friends or relatives.

In the study by Williams and colleagues,290

15 people referred to a community mental health team were interviewed both before andafter their consultation to ascertain if they weresatisfied with the treatment they had been given. It is not fully detailed in the text how this samplewas selected and therefore the possibility of abiased sample cannot be discounted. Carefulconsideration was, however, given to the natureand setting of the interviews. It was felt that thepeople would feel more comfortable being inter-viewed in their own homes. The authors stated that interviews conducted outside of the medicalsetting would help to ensure that they would not take the form of a medical consultation.Participants were encouraged to talk about theirexperiences in lay terms; in the terminology of our review, this may also have helped to ensurethat the questioning was acceptable to theparticipants which, in turn, is likely to increasereliability and validity. The analysis stage is detailed in the text. A grounded theory approachwas adopted, and new patients recruited in orderto provide further explanations for informationdeemed to be relevant to the study’s aims. Thesecond round of interviews was presented as a way of checking the validity of the study becauseparticipants were given the opportunity to providefeedback concerning the interviewer’s perceptionof what they had stated in the first round. Thestudy used a small sample of patients, whichensured that each case could be examined indetail, but has implications for the generalisabilityof the study.

Crabtree and Miller291 found that long interviewscan take between 1 and 6 hours to complete.Between three and six OE questions are asked and are designed to draw out long, detailedanswers. The development of the questionnairewas carefully conducted in this study. Issues ofvalidity and reliability were considered at all stages;for example, all of the members of the researchteam listed their beliefs, derived from personalexperience, about pain. This was done in order to understand biases that might be introduced.Interestingly, at this stage the authors mention that biases cannot be eliminated but can berecognised and used effectively in the researchproject. The authors recognised that it wasimpossible for the researchers to be entirelyobjective, but took steps to comprehend and utilise the individual ideas brought in by theresearchers involved. During the interviews anumber of ‘floating’ and ‘planned’ prompts were used to explore further the responses of the interviewees. The authors stated that “if the


43

long interview is successful, the outcome will be an interpretation that simultaneously reflects theparticipant’s reality and has generalisability totheory”. Pilot interviews were conducted first to modify and improve the questionnaire, andsecondly to improve interviewer performance interms of reducing intrusiveness and distortion. A purposive sample was obtained to ensure thatinterviewers and interviewees were strangers, thatrespondents did not have a specialised knowledgeof the topic under discussion and that there was aspread of age, gender and status. Four physiciansand four patients with recent experiences of painwere selected. Three researchers read the trans-cripts and identified utterances, thus enhancingvalidity and reliability. These were then systematic-ally compared and summarised into a quantitativestyle code-book. Each researcher identifiedemerging themes. The process of comparing the physicians and patients with one another wasthen carried out, although it was not completed at the time of publication of this review. It is stated that the conclusions drawn here weregeneralisable and transferable to other settings. It was concluded that this is a lengthy process; for example, 256 hours were required to analyseeight interviews. It is stated, however, that in totalthe project should be able to be completed withina year, which is comparable to “many primary care epidemiological studies” and therefore anacceptable time scale. Additional costs includedinterviewer time, transcription time and use of a computer.

One-to-one interviews have been compared with other forms of interviewing. Wilson andcolleagues292 discussed the merits and pitfalls ofusing face-to-face and telephone interviews. Thepaper highlights many of the issues that needconsideration when embarking on a one-to-oneinterview. A study looking at continence care wasundertaken. A pilot study used along with the five considerations presented by De Vaus (cited in Wilson and colleagues292; see appendix 3)revealed that telephone interviews would be themost appropriate technique for use in the mainstudy. Considerations included response rates,representativeness of the sample, design of theinterview schedule, anonymity and cost. From apractical point of view telephone interviews wereperceived to have several advantages, includingproviding greater interviewer safety. Indeed, it was expressed in the text that on some occasionsthe interviewee would become ‘over familiar’ with the interviewers who were relieved to be con-ducting the interview over the telephone. Also, it was found that four to five, and sometimes six,

interviews could be conducted per day, comparedwith two or three face-to-face interviews. This canhelp to reduce interviewer travel and interviewtime costs. Groves and Kahn (in Baker8) found thateach interview worked out to be less than half the cost if it was conducted over the telephone. Thereduction in time was seen as being advantageousbecause fatigue may be an issue if the interview is very lengthy, which may affect the quality ofanswers, which in turn may affect reliability andvalidity. Such interviews are more impersonal than face-to-face interviews and therefore therespondent may be more willing to talk aboutpersonal or embarrassing issues. It was perceivedby Groves and Kahn, however, that respondentsmay feel slightly rushed and less relaxed than if the interview was face-to-face.8

Telephone interviewing might use slightly differentlanguage and more clarification or explanationmay be required because there is no opportunityto present visual cues.8,9 In addition to this,responses may be shorter and less informationdivulged over the telephone because probing iseasier in the face-to-face situation.8 This maychallenge the overall reliability and validity of the technique.

It has been proposed by Foster287 that using theInternet and email services may be a practicalalternative to face-to-face interviews. This tech-nique was used to look into how teachers andlecturers plan and design their courses. Fosterstressed that this is a very new technique and one that needs much consideration and carefulthought before employing it as a research method. The main advantage of such a medium is that time, and therefore money, can be savedowing to the elimination of transcribing interviewtapes. The information arrives in a form that can already be recognised and analysed by thecomputer, meaning that transcription is notnecessary. In addition, this method can belogistically advantageous. There is no need for time to be spent travelling to participants’ homesor offices, participants can reply at their ownconvenience and can decide whether to take part or not without feeling pressured in any way.However, there are a number of disadvantages tothe method. There is a risk of it being viewed as‘junk’ or ‘nuisance’ mail, and only those with aspecific interest in the subject matter may reply;this has implications for acceptability, validity,reliability and generalisability. In addition, there is no opportunity for the interviewer to probe new issues coming up, or to clarify points mis-understood by the respondent. Identifying


44

suitable audiences is also a problem, and using this method alone could miss a large portion ofimportant opinion; therefore it is concluded thatthis method is unproven.

The dyadic interviewGeneral methodological issuesSohier293 argued that this type of interviewmaintains objectivity, enhances the credibility of the data, and therefore the reliability, and is also ethically sound. He argued that if two closelyrelated people are interviewed together then more detailed and contrasted information can be liberated. Information can be clarified orcontradicted, thereby providing a more reliable andvalid set of results. He noted that if the intervieweebecomes overwhelmed with emotion because of the difficult topics under discussion, the secondparticipant can then take on the role of comforterand can take over the dialogue for a period of time. This aids the interviewer as it enables him or her to “maintain the research posture” and canenhance the quality of the data. The situation mayoccur where both parties become very upset andemotional. In this instance the interviewer mustdecide if it is appropriate and ethical to continue,or to close the interview. This highlights again thatthe interviewer must be highly skilled. Anotherdisadvantage suggested by Sohier is that sometimespeople may not want to reveal too much inform-ation in front of those very close to them. Hestressed that confidentiality must be assured andthat the interviewer must remain compassionateand non-judgmental throughout the process.

Case study analysisGeneral methodological issuesGiven that case study analysis represents a newmethod of analysing qualitative data, there is very little methodological work considering thistechnique. However, it has been noted that whilstcase studies have the advantage of providing a set of extremely rich data, this is at the expense of generalisability.569

Delphi techniqueGeneral methodological issuesThe Delphi technique was reviewed in a recentHTA report by Murphy and colleagues.570 Anadvantage of the Delphi technique is that it is often difficult to organise meetings with pro-fessionals or ‘informed individuals’.570 There is no need for participants to meet up in onelocation and therefore a wider scope of opinioncan be gained because the logistical problems of organising meetings are eliminated. Also, as the participants are consulted on a number of

occasions there is the opportunity for statementsand suggestions to be changed or withdrawn as a period of ‘considered thought’ is allowed. Theanonymity of the process could be extremelyadvantageous. It is possible that more controversialissues could be raised if individual identity isprotected. The criterion of objectivity is rarelymentioned in the literature. However, Williams and Webb571 stated that researcher bias can occur when analysing the findings. Murphy andcolleagues highlighted that there has been littleresearch in assessing the quality of the Delphimethod.570 However, in several of the studiesmentioned below an attempt has been made toevaluate the technique and, in some cases, thestudy itself.

The technique also has a number of disadvan-tages. Response rate can be a problem and oftendecreases as the rounds progress,571 leaving themethod open to response bias. The Delphitechnique is a consensus method and this maymean that some important ideas are eliminated.303

These issues affect both validity and reliability.

Examples of the application of the techniqueIn the study by Guest and colleagues,294 they cited Le Pen and colleagues as stating that this is frequently used in healthcare assessment and is “a useful tool in situations where data are notavailable, but resource allocation decisions need to be made”.

Endacott and colleagues,295 in their study oncritically ill children, carefully evaluated andreflected upon the limitations of both the tech-nique and the application of it. It was found that,although attempts were made to minimise theworkload of paediatric intensive care sisters, it wasstill a considerable amount, thereby affecting theacceptability to respondents. This may also haveaffected response rates and therefore gained onlythe opinions of a limited number of professionals.In 1994, it was said that there is no evidence ofreliability for this technique.571 In the applicationpresented here, however, extensive efforts weremade to maximise validity and reliability bypiloting each round on nurses who worked with critically ill children.

Harrington296 found in his study on seniorpractitioners on research priorities in occupationalhealth that, although a degree of consensus wasreached, it was remarked that perhaps asking sucha specific group to comment could limit the scopeof opinion; thus, the representativeness of thepanel was questioned. It is also stated that the


45

types of professionals included here may beconcerned with strategic as opposed to practicalissues, limiting the study in terms of reliability and validity. Charlton and colleagues,297 in theirstudy on spending priorities of different healthprofessionals, found evidence to suggest thatmembers of different professional groups do notchange their opinions after obtaining feedbackrelating to other groups, therefore strengtheningthe argument for having a multidisciplinary teaminvolved. A flaw in the validity of the method canbe seen when one of those involved was asked tocomment on the results: she felt that her prioritiesstated in the questionnaire were not necessarily herchoices for priorities in research, which suggestedthat the technique required modification.296

One study carried out by Roberts and colleagues298

used a combination of methods and compared theresponses of consultant geriatricians and patients.Using the Delphi method, 89 geriatricians wereasked to formulate a list of performance measures.Also, 44 day-hospital patients were interviewed andasked to state those factors that they considered to be important to them. It concluded that, asdifferences were found between the priorities ofthe two groups, it was advisable to involve differentinterested parties when making priority-settingdecisions. The information was gathered from thepatients by interview; while this could have affectedthe sample size and therefore the generalisabilityof the study, it was seen as advantageous but alsoinfluential to have a researcher present for thepatients should they have any queries.

When considering the criterion of cost it is said to be a relatively inexpensive method of gaining alarge number of responses.9,571 This is especiallytrue because questions are distributed by post,which additionally ensures anonymity from otherparticipants in the study. This is stressed as beingan advantage of the Delphi method as the issue of a few dominant individuals taking over dis-cussions will be avoided.297 However, Charlton and colleagues297 experienced non-response as aresult of minimising costs. It was decided that inorder to save money that no initial meeting beheld explaining the study to those targeted; thismeant that there was a lack of knowledge aboutthe objectives of the study and in turn led to alarge number of invitees not participating.

Hadorn and Holmes299 used Delphi to asksurgeons, clinicians and “relevant specialists” their opinions on elective surgical procedures.However, little information was provided in thearticle on the inclusion criteria and numbers

involved. Also, to include social factors in thecriteria two public hearings were held. Members of the public were selected randomly but it is notmade explicit how this was done, how many peoplewere involved and how these factors would affectthe suggestions made and points raised at thesemeetings. These factors combined make it verydifficult to evaluate the use of Delphi in thisapplication as far as reliability, validity andgeneralisability are concerned.

Gabbay and Francis also conducted a study usingmore than one group.300 The Delphi technique wasapplied to ask general surgeons and anaesthetiststheir views on the maximum potential for daysurgery. There were problems with generalisabilitybecause the surgeons involved were reluctant tocomment on subspecialist areas, questioning thereliability of these parts of the study. Cost is notmentioned but the method appeared to beacceptable to the respondents as a largeproportion participated in all stages.

Wilson and Kerr301 asked 353 bioethics societymembers and their designates about importantsocial values related to healthcare. However,generalisability is questionable because the sampleobtained was not representative as it consisted ofwell-educated, affluent members of society only.Missing out those in lower socio-economic groupsand those with less knowledge of healthcareseriously limits the study. Additionally, the tech-nique involved four stages. The fourth stage wassent to non-responders to the initial phases andwas used to establish reliability and generalisabilityto the results obtained earlier. This forth stageyielded a poor response rate and thereforedetailed comparisons were not made. Thomsonand Ponder302 used the Delphi technique todevelop a survey technique in order to identifypriorities for the Texas Society of Allied HealthProfessionals, along with other state affiliates. A panel of five people chosen by the researchersnominated 21 health professionals to take part inthe development of the survey technique. Thisnon-random selection of participants could haveresulted in recruiting a biased sample. Time-consuming efforts were made to trace those whodid not respond and the method was piloted toestablish a Delphi question for use in the mainstudy that was neither too vague, nor too narrow.The statements made that were subsequentlyeliminated were reported and reasons for theirexclusion provided. Based on the researchers’experience of applying Delphi, a list of sevensuggestions was provided to guide the futuredevelopment of a survey technique.


46

One study that involved patients in the Delphi process was conducted by Gallagher and colleagues303 who looked at policy priorities for improving care for diabetic patients. In all, 28 ‘experts’ were questioned, including patientswho were already receiving treatment for diabetes.A high degree of consensus was found. Reliabilitywas addressed by using clear selection criteria and attempting to contact non-responders.Researcher bias was minimised in the collationstage by all three authors categorising responsesindependently. The researchers were aware of the potential loss of issues through centralising opinions.

Another study that attempted to incorporate the views of the public was carried out by Burnsand colleagues.304 Representatives of relevantorganisations were invited to contribute, althoughit is unclear how many participated and in whatcapacity. The study concerned building consensuson the care of those with physical disabilities.Reliability checks were made for correlation ofresults for rounds two and three by comparingissue-by-issue point scores in these two rounds. It was stated that these showed a high level ofcorrelation and were therefore very reliable. A second reliability check was undertaken bylooking at the responses of people who joined the study at the beginning (stage 1) and those who joined later on (stage 2). Again, no signifi-cant differences were identified between thesegroups and the authors stated that the “addition of participants did not significantly alter the study”.

Kastein and colleagues305 looked specifically at theissue of reliability of Delphi in a primary health-care setting in The Netherlands. They conducted a study to develop evaluation criteria for the per-formance of GPs when they are consulted aboutabdominal pain. A secondary objective was to look at the reliability of their application of thetechnique. They deal with “minimising situationspecific biases” (this terminology refers to what we have indicated as quixotic reliability) by stand-ardising recruitment procedure. They also soughtto minimise “person specific biases” by carefullyselecting their sample. Both family physicians and specialists were invited to participate. How-ever, as previously discussed, merely obtaining areliable sample does not ensure the reliability of the technique as a whole. Diachronic andsynchronic reliability were not dealt with, as well as other issues concerning validity, objectivity ofresearchers, generalisability and cost. This study is, however, one of the few that attempts explicit evaluation.

Complaints proceduresGeneral methodological issuesThis type of exercise, although useful in identifyingcomplaints and comments, will only incorporatethe views of those with particularly strong opinionsand those with especially negative experiences, sothat generalising the results of this type of study is extremely difficult, if not impossible.572

A number of additional studies looked at using this method for obtaining public preference.Hemenway and Killen573 suggested that using thecomplaints of dissatisfied patients can be used asan inexpensive and effective way of discoveringinformation that can be used by policy-makers, but warned that this information should be used wisely because all complaints are notnecessarily valid.

Reid and colleagues335 discussed the role of thecomplaints procedure within the NHS with a viewto quality assurance. The ‘Oregon experiment’574

looks at involving the public in healthcare ration-ing. Using complaints can indicate the strength of feeling concerning a particular treatment orintervention, but again, these complaints may be invalid.

Dean and colleagues575 highlighted that there are two different types of complaint. The firstconcerns dissatisfaction with a service that hasbeen provided and the second is concerned with refusal of treatment or intervention. Usingcomplaints as a rationing technique was describedas being dangerous because of the possibility that“services will go disproportionately to those whoare most determined, articulate or litigious” (p. 343), which raises serious questions of equity in healthcare.

Group approaches

Focus groupsGeneral methodological issuesFocus groups are sometimes persceived to be an easy method.576 It is demonstrated here,however, that it can be a very challenging method to use and needs careful considerationand planning. A number of general principlesmust be considered when undertaking this method of research. Having a trainedmoderator/facilitator is vital for good focus group research.307 The quality of informationgathered can be determined by the expertise of the facilitator/moderator;576 for example,expertise in skilled questioning and careful


47

probing. The facilitator is given the choice of how much control to have, but a certain amount of free-flow is needed and therefore carefulattention needs to be paid to the selection of themoderator. It is not always the moderator’s role tofind common ground but to identify different andperhaps conflicting views from group members.577

If the facilitator is not suitable, the results of thegroup may be misinterpreted and inappropriatetopics discussed331 and bias may be introduced;Festervand highlighted that this can affect reli-ability.306 Participants are sometimes asked to take part in games: sorting vignette, “speciallydesigned board games, ‘cake cutting’ or coloureddisc games, for example, in groups about healthpriorities and allocation of resources”.9 This indi-cates the use of quantitative techniques, as well as qualitative techniques, within focus groups.Another important consideration is the setting. It is vital that participants feel relaxed, and that the room is comfortable. Additionally, it must becarefully thought out as to whether it is advan-tageous for the group to be friends or strangersbecause this can have implications on the depthand kind of information disclosed. These issues will be discussed using a number of examples below.

A major advantage is that group interaction can be observed.307 The interactions differ bywhether or not participants know each other; for example, people who work together or see one another on a daily basis, may feel inhibitedwhen discussing personal or work-related issues.Group size can have an effect: Tang and Davis578

listed the following critical factors in determiningoptimal focus group size: aims of the study; thenumber of question asked; the allotted time foreach question; the format of the focus groupsession and the duration of the session.578

Combined group effort produces a wider range of information and ideas to be expressed com-pared with one-to-one interviews.306 Groupdiscussions can be stimulating and members can find security in numbers and will be morewilling to divulge information. Also, as participantsare not required to answer each question, theresponses are likely to be more meaningful. Inaddition to this, one comment can stimulate others into thinking of related ideas producing a snowballing effect and therefore perspectives that may otherwise be missed can be explored.579

This technique gives people the opportunity tofocus upon common rather than individualinterests.311 The discussion process in itself enables participants to form opinions, armed with fresh knowledge. People often do not know

what they think about certain issues, especiallythose in which they have little prior knowledge.311

Additionally, they are perceived as being easy to conduct309 and yield quick results.310 In realitythis is not the case, conducting a focus group takes a lot of planning and organisation, andrequires skilled professionals to conduct them. In comparison with individual interviews, larger sample sizes can be used and it is possible to explore issues in depth and pursueunanticipated areas.310

There are a number of disadvantages associatedwith focus groups. Although attitudes can beexpressed, the extent of these attitudes is diffi-cult to gauge.576 The logistics of arranging andcarrying out a focus group can be tricky: gettingthe participants together in one place at one timecan present problems.307 Focus groups can beconducted quickly and more cost-effectively thanother methods such as surveys, which means that inappropriate topics may be explored.306

However, relative to a questionnaire survey it is still a costly exercise because transcription andreporting are time-consuming and thus labourintensive.379,568 Individual responses and opinionscan be influenced by the rest of the group.309

Some people may respond in a way deemed to besocially acceptable, therefore swaying the resultsand affecting validity. This can be especially true if investigating a sensitive or controversial topic.379

People may find it unacceptable to discuss thesetypes of issues in front of or within a group ofpeople, and therefore not reveal the private issuesthat they would on a one-to-one basis.307 Com-pared with one-to-one interviews, there is less time for each person to talk and thereby to reveal their preferences.307,576 In addition to this,some ideas may not be expressed because ofconcern that others will perceive them as sociallyunacceptable, racist or sexist. Participants may also experience pressure towards consensus andunanimity.580 It is recommended that, to ensurereliability, two researchers independently code and analyse the transcripts, compare them, andagree upon important themes. This can also help to ensure that objectivity of the researcher is achieved.9

Because of the intensive nature of the method,sample sizes are small and therefore findings may not be generalisable. An aid to overcomingthis is the careful selection of members that canrepresent the study population as closely aspossible. All too often, however, the sample group proves not to be representative.306


48

Additionally, issues of self-selection or self-exclusion need to be addressed. It is likely thatthose who are particularly interested in the topicbeing studied have particularly strong opinions, or those who are extrovert are more likely tovolunteer than those who are shy or withdrawn.

Methodological issues involved in using focusgroups in studies in healthcare suggest that insome instances it is appropriate to ‘break the rules’of conducting focus groups, emphasising that eachtopic needs to be individually considered and thatit is difficult to adhere to a predefined formula.313

Examples of the application of the techniqueSomerset Health Authority used focus groups toinvolve the public in priority setting.311 Eight focusgroups of 12 people each met three times a year.Professional recruiters were employed to gain arepresentative sample. The CHC members wereused to test the effectiveness of focus groupdiscussions. Groups discussed real issues ratherthan hypothetical ones; for example, the questionof allowing and funding treatments outside thecounty was addressed. Background information was supplied and options of a figure to be agreedupon suggested. A briefing paper was also sent to participants in advance of each meeting anddiscussion was encouraged with friends and familymembers – something that was undertaken bysome group members. The sample population wasdemographically representative of the communitybut with a slight bias towards those with a greaterexperience of using the health service. The authorsargued that using professional recruiters made thefocus group representative and that trained andexperienced group facilitators led to the creationof a “richer bank of information than could beobtained through a more conventional structuredsurvey”.311 The analysis was not made explicitabout, for example, who was involved and whatmethods were used. Had this been done thevalidity and reliability of the technique could have been further established.

Kuder and Roeder312 investigated public attitudestowards age-based priority setting in America. Fivegroups were held and separated into homogenousgroups based upon age and socio-economic status.Gender and race were not considered in theselection process, which could therefore haveinfluenced the results; for example reference torace was avoided in the groups. Two researcherswho identified and agreed upon general themescarried out the analysis of the transcripts.Additionally, the coding process was tested forreliability by asking another member of staff to

code one transcript. It was found that participantsof all ages were reluctant to endorse a policy thatlimits care to the elderly, and were unable todecide the extent to which the government shouldbe responsible for subsidising treatment costs.

In contradiction to what Morgan309 and Krueger310

said, there are situations where it is appropriate to use focus groups to explore sensitive orembarrassing issues. This is demonstrated in the Cohen and Garrett313 study where client/worker relationships in residential mental healthsettings were looked at. During the focus groupsthe facilitator found that issues such as physicalabuse and suicide attempts could be discussedamongst group members. They would support one another and offer information on similarexperiences. It was important for the facilitator togain trust before proceeding, and it was reportedthat this trusting relationship enabled in-depthdata to be gathered. This again highlights thenecessity for the moderator to be an experiencedgroup interviewer.

Kitzinger314 also investigated a sensitive topic: 52 groups were conducted, comprising 351 people,to investigate attitudes towards and knowledge ofAIDS. The groups were selected from pre-existingsocial or work-based groups. These ‘naturalclusters’ were chosen for a number of reasons.First, they were more likely to challenge oneanother because they had knowledge of eachother’s circumstances and, secondly, discussionscould take place in social contexts that could aid in increasing the reliability of results. Thefacilitator engaged the participants in exercisessuch as card games to encourage everyone to talk, which helped the shyer group members tocontribute at the start of the session, therebyboosting confidence for the rest of the discussion.Kitzinger was particularly concerned with maxi-mising group interaction stating that this enabledunexpected issues to be covered through sharingcommon experiences and arguing over issues, so that both similarities and differences betweenmembers could be examined. The purpose of thestudy was to find diversity rather than to establishrepresentativeness. Results, therefore, may not begeneralisable, but nevertheless the study providesthe reader with an in-depth insight into the viewsof different groups. Because of the size of thepopulation this study must have been extremelycostly, which is one of the main drawbacks of this type of data collection.

These two studies can be used to highlight that in some situations discussing sensitive or


49

embarrassing topics is acceptable to participantsand that, again contrary to Morgan309 andKrueger,310 rich, in-depth data can be extracted.This may be because group members feel thattheir participation is legitimised and valued. It ispossible that people can feel intimidated if askedto talk in front of others who are perceived to beon a higher social or professional level.

Ward and colleagues315 found that certain topicsare not suitable for discussion in focus groups. This was found to be the case when male attitudestowards vasectomy in Latin America were sought. It was unacceptable for men to admit that they had had a vasectomy as it could be associated with being “less of a man” and therefore inappro-priate for public discussion. This can be used toillustrate that the way in which the question isasked can be very important to the reliability and validity of results. It was found that responseswere provided to questions about whether having a vasectomy would affect their status in the com-munity, rather than being asked directly aboutpersonal experiences.

In the study by Carey and Smith,316 it was foundhere that using focus groups was unacceptable.This was because topics such as sexual orientationand drug and alcohol use needed to be addressed.Discussing these topics with colleagues was notpossible for the members because disciplinaryaction might have been taken.

Another example where the ‘rules’ of conductingfocus groups were broken is in a study carried out by Powell and colleagues.317 Both users andproviders of mental health services were asked todiscuss provision of and access to these services.The groups were pre-existing, and this was seen as advantageous as the familiarity of memberswould provide a supportive atmosphere. Staff fromdifferent professional backgrounds were chosen inorder to “maximise the discussion of differentperspectives” and were of an equivalent level so as not to inhibit contributions by staff who may feel intimidated by the presence of higher levelmembers. The groups were self-selecting, and noattempt was made to stratify for age, sex, race,occupation or length of contact with the service.The facilitator asked broad, OE questions andattempted not to lead the discussion. Reliabilitywas enhanced at the analysis stage by using twofacilitators and comparing their analyses. Theauthors stated that the focus group method wasnot used to its full potential; it was used here toprovide validity for questions already being askedof users and providers in questionnaire format.

It was recognised by the authors that, aside fromthe limited generalisability of focus groups ingeneral, this study has a circumscribed generalis-ability because the data collected only relates tothe personal experiences of those in attendance,and that both the user and provider groups werenot representative. Also, because the groups werepre-existing it is likely that they were composed of the more confident, vocal people, and wouldtherefore miss the important inclusion of thequieter and perhaps more frequent users of theservice. These limitations are recognised andacknowledged by the authors who state that the validity is not affected as they were merelyattempting to validate pre-existing questionnairesand to generate new ideas for questionnaires.

It has been previously stated that interactionbetween participants is a key factor in focus groups and can provide very rich and useful data.Wilkinson318 looked at women diagnosed withbreast cancer and was interested to learn abouttheir experience of treatments and surgery. It was found that the women could talk openly about very personal experiences and they evenjoked about common encounters, meaning that as well as gathering data the focus group was also a therapeutic exercise for the participants.

Stevens319 carried out an additional piece of workthat very much focused upon the interactionbetween individuals. Stevens looked at gatheringinformation at the aggregate level. The focusgroup was used as a method of validating what wassaid in individual interviews conducted previously.Stevens states that not only can focus groups beused to gain in-depth insights but can be directedat segments of the population, which may havebeen under-served. A total of 13 lesbian womenwere asked to share their health experiences. It was found that during the discussions the womenwere able to talk freely about their experiences and could validate one another’s concerns. As OEquestions were asked the women had the freedomof directing the discussion and could talk aboutissues that were important to them; this is stated inthe text as validating the data. Individual interviewsalso took place, demonstrating a triangulation ofmethods and enabled comparisons to be madebetween the different data collection methods. The group facilitator carried out the analysis andtherefore may have been introduced observer bias, although a familiarity with the results andlanguage used enabled themes and discrepanciesto be dealt with. Again, the subject matter wasconsidered to be acceptable to the respondents;this is evident because of the women volunteering


50

personal information and the free flowing natureof the discussions. This study was not intended tobe generalisable to a wider population, rather toelicit responses from a specific group so as toincorporate their views on health topics that mayhave been missed during previous research. Thewomen were all, however, very well educated andtherefore selection bias may have influenced theresults even though attempts were made to includewomen from different ethnic backgrounds.

In the study by Keller and colleagues,320 22 seniorcitizens were invited to attend focus groups toinvestigate their healthcare needs and beliefs. Inaddition, 16 22–40-year-olds were asked to expresstheir opinions on the topic. The facilitator of the discussions used a set of nine OE questionsdefined before the sessions commenced. This is in contrast with the previous study319 in that theissues to be discussed were predefined, rather than being led by the participants themselves. The sessions were all recorded and subsequentlytranscribed, although again it is unclear whocarried out this task, and indeed if more than one researcher was involved. Follow-up calls weremade to participants to give the opportunity forindividuals to provide feedback and to validatewhat they had previously stated.

Smith and West321 also sought the opinions ofsenior citizens. In all, 124 older people were asked about their experiences and expectations of health. All of the groups were facilitated andtranscripts were analysed by two researchers whoensured that all relevant topics were covered inadequate detail. The focus group was found to be an acceptable method that raised someimportant issues in the provision of care to older people.

Bradley and colleagues322 conducted a study whereattitudes towards England’s health strategy wereexplored. Here, 173 people took part in 24 focusgroups. Again the groups were organised in ahomogeneous manner, although the reasons forthis are not stated. Reasons for exclusion of certaincitizens were provided; that is, those with a historyof HIV or AIDS or registered injecting drug userswere not invited to participate as lay perspectiveswere sought. Including those people with aspecialised knowledge may have swayed the results.In addition to this, unnecessary distress may havebeen caused for limited benefit to the study. Issuescovered in the different groups included coronaryheart disease and stroke, accidents, mental illnessand sexual health. The validity of the analysis stage was established because two independent

qualitative researchers carried it out. The cost ofeach 90-minute focus group is stated as being £365, taking into consideration recruitment,facilitating, recording, transcription and analysis.The generalisability of the findings is not men-tioned but it is concluded that the method wasinexpensive, that participants were able to graspthemes and that it was feasible and thereforeacceptable. The authors express that this is avaluable method, which should be utilised whengauging citizens’ opinions on healthcare.

The effect of discussion and deliberation on thepublic’s views on priority setting was examined byDolan and colleagues.323 The aim of the researchwas to establish how much views had changed afterdeliberation. A total of 60 people were randomlyselected to take part in the groups that consisted of five to seven people; the samples were stratifiedto include people from all ages and a balance ofgender in each group. All participants were invited to attend two meetings, held 2 weeks apart, and were assessed as to whether their viewshad changed in the period of time between thebeginning of the first group and the end of thesecond one. The two researchers who ensured that all relevant issues were covered also con-ducted all the groups. It was stated that to ensureexternal validation the sessions were tape-recordedand transcribed, although it is unclear who carriedthis out. It was concluded from the study thatpeople’s opinions on healthcare issues changedconsiderably after the opportunity for discussionand deliberation had been provided, and thereforevalid results will be obtained after such discussion.The focus group method was compared withsurveys in the text and it was concluded that thesemethods generate similar results; however, in thisreview the finding is that this is not the case andthat the method used can have a significant effect on the results obtained.

Additionally, Ramirez and Shepperd324 found that a questionnaire presented to participants wasuseful on two counts: first, to gather demographicinformation on those taking part and, secondly, to prompt people to think about the issues to bediscussed, thereby allowing a period of consideredthought. Also, informing people that the inform-ation gathered during the groups would be used toimplement new health programmes was perceivedto be advantageous when recruiting people. It isstated that people may be more willing to take part if they feel that if what they say will be used to improve healthcare in their own area. Thisparticular group sought knowledge of and attitudestowards different risk factors associated with cancer


51

with a view to implement an intervention topromote healthier living in the local area.

The issue of analysing transcripts was dealt with by Weinberger and colleagues.325 The study wasconcerned with whether or not the rating system of transcripts was consistent amongst researchers.A total of 101 ischaemic heart disease patients and 29 physicians participated in focus groupdiscussions. Patients’ experiences of care and thephysician’s decision-making processes concerningtreatment procedures were addressed. The groupswere recorded and transcribed before asking sixhealth professionals to formulate a list of factorsmentioned as being involved in decision-makingfor cardiovascular disease. Three raters were asked to categorise these factors as being eitherminor or major, to ascertain if the raters wereconsistent in their judgements. Each transcript was rated by at least one physician and one non-physician to ensure that both clinical and non-clinical perspectives were considered. It wasconcluded that consistency in raters’ judgementswas difficult to achieve, even with trained raters,and this therefore seriously undermined thereliability and validity of focus groups. It wassuggested that it might be necessary to quanti-tatively analyse transcripts, although it was notedthat this might be an unpopular suggestion toqualitative researchers.

In conclusion to this section, it can be seen thatfocus groups have been used widely in healthcare;a selection of examples has been shown here.These show that the focus group is a particularlyuseful tool in discovering people’s opinions andattitudes towards their healthcare. The dataprovide explanations as to why people adoptcertain perspectives towards their healthcare, and is therefore appropriate for exploring priority-setting issues.

Concept mappingGeneral methodological issuesSouthern and colleagues326 note that this a usefuland acceptable way to gain group opinion andTrochim and Linton327 note that researchers haveno control over the final outcome, thus aidingobjectivity. However, Trochim and Linton also state that it is a time-consuming and costly exercisebut they also stressed that, with the development of new technologies, these costs can be reduced.They also stated that more work is need in thisarea to establish strengths and weaknesses of the process. Given that the technique involvesfeedback to respondents, there is the opportunity

for respondent validation, therefore increasing theoverall validity of the process.

Citizens’ juriesGeneral methodological issuesIt has been argued that citizens’ juries offerinformation, time, scrutiny, deliberation andindependence.331 A major strength is that informeddeliberation can occur.331 Lay people are given the opportunity to listen to, absorb and deliberateover information coming from several angles andfrom a range of professionals. People’s rationaldeliberation is relied upon as opposed to a knee-jerk reaction.581 It is felt that the deliberationprocess increases validity and reliability and thatthe method is particularly suited to difficult andcomplex questions.331 Some jurors have expressed,however, that it would be beneficial for the timeavailable for deliberation to be extended andperhaps a break would enable them to process the vast amount of information received.333 Inaddition, strengths are “clear aims and role forjurors, a mechanism for implementing juryrecommendations, and the incorporation offeatures such as information provision, time toquestion witnesses and deliberate, and a degree of independence and authority”. These featuresindicate that the method is an acceptable one tothe participants.

Citizens’ juries have, however, recently beencriticised as being “morally and democraticallyirrelevant”.581 Price argued that individual issues or case studies cannot be generalised to otherissues and that it is incorrect to assume thatprinciples drawn up for one situation can then be universally applied.581 In addition, Price believes that participants are asked to perform an impossible task. They are, on one hand,expected to be neutral and impartial but, on the other hand, are tax-paying members of thecommunity “committed to family, friends, his or her city, or even the NHS”. Price continues,“Citizens are at once disinterested and committed.They contemplate broader issues and yet aremotivated by relatively narrow ones.”

Citizens’ juries can be a very time-consuming andtherefore expensive process, especially since ittakes a very limited number of people’s views intoaccount. The planning of the exercise is a veryimportant component and therefore adequatetime needs to be set aside.331 A 4-day jury can cost as much as £25,000 (in 1997); this includedexpenses such as recruitment, incentives for thejurors, venue, development of an agenda,recruitment of witnesses, moderators and


52

publication and dissemination costs. Clearly, this isa drawback; however, if carried out in a rigorousmanner, the cost can be justified because it obtainsa more informed opinion of the public than aquantitative survey.328

A further potential for bias or lack of objectivity in citizens’ juries is for the witnesses to manipulatethe jury. In one study a person with no knowledgeof the subject matter was selected as moderator toensure that the deliberations were as objective aspossible.311 A further disadvantage is that onlysome issues can be covered. Citizens’ juries havebeen shown to be good for choosing between two options and solving dilemmas, but not wherethere is a need to develop detailed plans or discusshypothetical situations.331,332 They can deal onlywith problems that can be addressed by a decisionand a set of recommendations at a particular pointin time329 and are unable to look at problems that need frequent review.582 Jurors do not set the agenda or are not able to bring in additionalissues as they see fit, which could potentially alterthe outcome.329 Currently, citizens’ juries are notused as a forum for joint discussion with thepriority decision-makers. This means that theprocess does not allow for members of the jury to meet and discuss relevant issues with the policy-makers themselves.329

Examples of the application of the techniqueCitizens’ juries are a relatively new way of findingout the public’s opinion on priority setting. Aseries of five pilot juries was carried out in 1996.One took place in Cambridge and Huntington;two in Kensington, Chelsea and Westminster(KCW); and one each in Walsall and Luton. Three additional juries sponsored by The King’sFund followed these up in Sunderland; EastSussex, Brighton and Hove; and Buckinghamshire.Each of these juries discussed different issuesconcerning healthcare policy and healthcarerationing.328 The first group to use it as a researchtool was the Institute for Public Policy Research(IPPR) in 1996 with Cambridge and HuntingtonHealth Authority.328,331,332

The citizens’ jury in Cambridge and HuntingtonHealth Authority aimed to (1) gather informationon how the jury members answered questionsrelating to priority-setting issues; and (2) evaluatethe process of citizens’ juries as a method ofeliciting public opinion. The 16 jurors wererecruited using stratified random sampling to get a representative sample.332 They were asked to address questions about whether the publicshould be involved in priority setting, the criteria

that should be used to make decisions abouthealthcare and who should set these priorities.331

Over a period of 4 days jurors heard informationfrom expert witnesses and were given theopportunity to question and cross-examine them and could deliberate over their decisions. By asking the jurors to fill out questionnaires both before and after the process, information was obtained about the successes and failures of the technique.332 One problem noted here is that one or two highly knowledgeable and/orvery vocal members of the group can limit theopportunity for everybody to express their opinion, a problem to be found in a number of qualitative group-based methods such as focus groups and public meetings. On thisoccasion this difficulty was overcome by splittingthe jury into two smaller groups to give the quieter members more of a chance to speakwithout feeling intimidated.

Kensington, Chelsea and Westminster HealthAuthority held two pilot juries to debate the issueof how the quality of life of mental health patientscould be improved. The Walsall jury investigatedhow £600,000 could be best spent to improvepalliative care. Finally, the Luton jury were asked“how should citizens pay for health services in the future?”331 In The King’s Fund projects, theSunderland jury was asked about whether localpeople would accept certain treatments fromhealth professionals other than their GP. EastSussex, Brighton and Hove were asked to con-sider where women should be treated who havebeen diagnosed with gynaecological cancer. InBuckinghamshire, jurors were asked whether the health authority should fund treatment forback pain.333

These pilot studies were evaluated by Sang (cited in Davies and colleagues333). A number of comments were made about the effectivenessand practicalities of the process. It was ultimatelyconcluded that a group of citizens were able towork well together as a team and could appreciatethe complexities of a health policy issue. Thehealth authorities viewed the jurors’ recommend-ations positively and the jurors viewed the experi-ence positively. However, three complications were detailed. First, there is the concern that local people may have difficulty engaging with themajor policy issues facing the NHS; secondly, therole of health authorities in making these types of decisions is not yet clear; and finally, it is feltthat the more the issue of involving citizens indecision-making is investigated the more complexit becomes, and the more questions need


53

answering. Coote and Lenaghan331 concluded thatprovided the question being asked is “prepared inan appropriate and manageable form” then theparticipants are willing and able to make sensiblejudgements about complex issues. It was realisedfrom the Luton jury that presenting four optionsof how to fund the NHS was too challenging and it was suggested that three or two would be moremanageable. They also felt that the jurors acted on behalf of community interests and not personalones and that jurors’ confidence grew throughoutthe proceedings.

The Welsh Institute for Health and Social Careadopted the citizens’ jury approach to investigatenew genetic technologies.334 The jury process wasconducted in a similar manner to those describedabove but had three major differences. First, amultinational pharmaceutical company sponsoredthe jury rather than a policy decision-making body. This meant that the recommendations made by the jury did not have to be adopted, and they did not have to offer explanations aboutwhy they would not do so. Secondly, the nature of the question being addressed meant that anunderstanding of material of a scientific nature was required. The jurors were provided witheducational material but this was rarely utilised,thereby questioning whether the participants fullyunderstood the issues. Thirdly, those selected totake part were to represent Wales and not a smalllocal community, but doubts were expressed as towhether this type of tool was appropriate for thislevel of policy recommendation. As in some of theprevious juries the issue of representation was aconcern for both the organisers and the jurorsthemselves. In this instance only one member had continued their education after reaching the age of 18, there were no participants fromethnic minorities or from a young age group and few of the jurors were in full-time employ-ment. This not only puts a question mark againstthe representativeness of the jury but also itsgeneralisability. The moderator chosen was highly experienced at working with small groupssuch as this. He attempted to ensure that allmembers took part in discussion, that timeschedules were adhered to and that outcomes of discussion were recorded. It is not made clear if he had specialised knowledge of the subject matter and therefore the extent of hisobjectivity cannot be commented upon. It wouldbe improved, however, by including a secondmoderator. This jury highlighted a number ofother issues relating to the potential for bias.There was a concern that some of the witnesseswere introduced using their scientific titles but

that other non-scientific members, who would beof equal importance, were referred to by nameonly. One lay witness felt that this undermined herstatus and felt “down-graded”. Additionally, muchconsideration should be given to the ordering ofwitnesses. Here one medical witness appeared both at the very beginning and at the very end of the process, which could have had implicationsfor validity and reliability. The participants andauthors viewed it as a positive experience but felt that “the primary justification for using thecitizens’ jury approach was educational rather than participative”. They underlined the notionthat some people feel uncomfortable about beinginvolved in these types of decisions but yet felt that it was useful as an educational tool. Theauthors concluded that “as yet there is littleevidence that the recommendations are more than a one-off snapshot view produced by a small, but enthusiastic, group of inhabitants of the principality”.

Since the technique was developed it has becomeincreasingly popular as a tool for aiding decision-making. The Portsmouth jury, held in 1997,addressed the question of what are the mostimportant criteria for setting spending priorities.328

To help the jury to test their criteria they weregiven descriptions of four treatments and asked toallocate a set budget of £80,000 (as such this canbe seen as a variation of the allocation of pointsmethod within a citizens’ jury). The Nottinghamjury was presented with the question: Should non-clinical factors be taken into account whenprioritising NHS resources?328 Lewisham held ajury asking the question: What can be done toreduce harm to the community and individualsfrom drugs?335

Consensus panelsGeneral methodological issuesAs this method is similar to the citizens’ jury, thesame methodological issues of discussion leaders,selection methods and deliberation need to beconsidered. An advantage over citizens’ jurieswould be the lower cost of the exercise, which may then make it a more practical and acceptablemethod to use.

Examples of the application of the techniqueA UK study conducted by Coulter and colleagues340

investigated the effect that panel composition hadon the ratings for use of spinal manipulations forlower back pain – a methodological issue withimplications for validity, reliability and generalis-ability. One panel consisted of a multidisciplinaryteam and the second panel consisted only of


54

chiropractic physicians. It was concluded that amultidisciplinary team was more likely to rate atreatment inappropriate than a team of profes-sionals who actually used the treatment or tech-nique. The three other studies did not detail howpanels were selected or highlight methodologicalissues. A study that did detail these issues wascarried out by Stronks and colleagues.336 Theyconducted a study in which the views of patients,the general public, GPs, specialists and healthinsurers were incorporated in a consultation toachieve a one-third cut in the healthcare expendi-ture. The panel members were given informationon ten scenarios of services that could be droppedor not funded. The members were asked to discuss these issues and decide on areas where cuts should be made. Panels were not selected to be representative of the population, but chosen to represent five distinct groups. Theauthors recognised that this would affect validitybut highlighted that it was not their aim to berepresentative but rather to gain insight into the arguments and concerns of different groups of people when considering priority issues. Afacilitator was appointed. It is not stated how this person was selected, and his or her personalviews could have influenced outcomes, therebyintroducing bias and questioning validity and objectivity.336

According to the authors this technique haslimited value because people had a lack ofknowledge relating to the issues they were asked to discuss and were not given enough time to absorb information and form well thoughtout opinions. However, this study was carried outusing particularly controversial subject matter. The conclusion drawn from the study was that “it is not clear that including all the differentactors in the decision-making process of prioritis-ation of health services will lead to more equitableor broadly supported outcomes or to better healthto the population”. It was established in this studythat healthcare professionals were more aware ofthe importance of equal access to health than were other groups, in turn suggesting that thepublic and patients are not best equipped to make these types of decisions when using theconsensus panel method.

Public meetingsGeneral methodological issuesPublic meetings have been shown to be an in-expensive and relatively quick method of obtaininginformation but can be severely criticised in thatonly those with a particular interest or who are very vocal about it will attend,583 and those

attending will differ from the general populationin demographic characteristics.341 Meetings areoften poorly attended and there is the potential for them to be dominated by specific interestgroups.328 Additionally, they have been criticisedfor addressing issues that are not of the greatestconcern to the public.583

Public meetings have often been used as a methodof making people aware of potential closures ofhealth facilities, therefore generating a feeling ofhostility to them.583 The more cynical may arguethat perhaps information-giving is the main aim of public meetings under the pretence of listeningto the public or that they are used to legitimisedecisions that have already been made.584

The volunteer bias questions the validity, reliabilityand generalisability of the method as the solemeans of eliciting public opinion. However, it isrecognised that they may still be useful if usedalongside other methods to gauge opinion.583

For public meetings to be as reliable and valid as possible it is recommended that they are wellpublicised, that there is easy access to all membersof the public and that all those in attendance areasked for their opinion. Jessop585 felt that for NHSmeetings these first two criteria are usually met but the third is neglected, and that gaining a mere impression of general mood is not acceptable or reliable.

Examples of the application of the techniqueGundry and Heberlein341 investigated hypothesessuggesting that these types of meetings are un-representative. They found that although it wastrue that people attending did differ in theirdemographic details the overall informationgained was the same as that from a survey of thegeneral population. It was also found that thosewith extreme opinions might well attend the meet-ings but that the opposite can also occur, in thatthose who attend can have stronger agreementwith the general public than those not attending.The authors argued that “public meetings may be a useful and valid tool for capturing a reason-ably accurate picture of public opinion on a variety of issues”. It is, however, recognised thatrepresentativeness cannot be assured unless asimultaneous survey is conducted alongside themeeting, which would then compromise the majorbenefits of low cost and ease of implementationthat public meetings can offer.

In the Broadbent study,342 issues surroundingadvertisement and access to public meetings in


55

her local area were explored. She found a lot ofvariation among the 12 trust boards she contacted.Seven were helpful and all the information re-quested was obtained from one telephone call; forthe other five, however, considerable difficulty wasexperienced in finding out when and where thenext meeting would be held. She also attended themeetings and found that a number of them were,first, difficult to find and, secondly, provided noopportunity for public participation. It was herrecommendation that each and every meeting beadequately advertised, directions of how to get tothe location provided, and board members beintroduced; those attending should receive awritten agenda and time should be allowed forcomments and questions.

A number of public meetings were set up and re-viewed in a study carried out by Gott and Warren.343

The community of the North Staffordshire districtformulated a Neighbourhood Health Forum, whichwas designed to increase local participation inhealth issues and used public meetings as a way tofind out the feelings of the community. To makepeople aware of the meetings and to increaseattendance, selected members of existing organis-ations, mainly voluntary groups, were personallyinvited. The meetings were also advertised, invitinganyone interested in being involved in local health-care decisions to attend. They were held in build-ings frequently visited by the public in an attempt to encourage participation. It was found that therewere a number of professionals in attendance whowere not directly working in the health sector. Thissuggested that attempts made to involve a diverserange of community members were successful,although no mention was made of the moremarginal groups such as unemployed people, ethnic minorities or people with disabilities.

This study showed that there is value in conductingthese types of meetings and that extensive stepswere taken to maximise attendance and therebygeneralisability. This is further illustrated with thesuggestion that information sheets, newsletters andpersonal invitations be distributed to households in the district. There is, however, still a dearth ofstudies that have evaluated methods of recordingand analysing the content of most methods ofpublic consultation.

Nominal group techniqueGeneral methodological issuesSimilar considerations concerning the facilitatorand participant selection are required for thenominal group technique as for the methodsdescribed above. There is a high potential for

selection bias348 and the composition of thosepeople taking part can have an effect on the finaloutcomes. There is also the potential for falseconsensus to be obtained, especially in situationswhere there is diversity of opinion on priorities.347

Redman and colleagues,349 in their study looking at priorities in breast cancer service provision inAustralia, note that the method appears to havestrong validity because there was agreement aboutpriorities amongst the different workshops. Addi-tionally, validity was demonstrated when resultswere presented to government agencies, clinicalspecialists and consumers who agreed with thepriorities identified. Redman and colleagues alsonote that the method is acceptable to participantswho were willing to participate and were satisfiedwith the process.


The systematic review identified a number ofqualitative techniques that have been, or could be,used for eliciting public preferences. Techniqueswere identified as individual or group-based.

One-to-one interviews are an extremely useful toolfor obtaining information from individuals abouttheir perspectives on health and healthcare. Theadvantages are that: the interviewer is present toclarify issues and provide verbal and visual clues; in many cases more personal information can bediscussed; each participant is provided with ade-quate time to fully express himself or herself. Themajor problems with this technique are that onlysmall samples can be examined because of thedetailed information required and that the method is very open to interviewer bias in respectof the topics discussed. Ways of minimising theseproblems are suggested in the literature and havebeen highlighted here. A number of new methodsoutlined here, namely telephone, email and dyadicinterview techniques, have not yet been used inpriority setting. Given their potential benefits, they worth investigating.

The Delphi technique is a flexible method thatenables the opinions and judgements of experts to be collected. Respondent bias is minimisedowing to the anonymity of the technique, whicheliminates peer group pressure, although theremay be a high level of drop-outs because it can be a three or four round process and thereforeonly the most committed may remain. Issues ofreliability, validity and generalisability can beaffected by the selection of the panel members. On the issue of objectivity, the method is liable


56

to researcher bias, and the minimisation of this is the responsibility of the individual researcher as is the case in other qualitative methods. Anadvantage to this method is that it is a cost-effectiveway of gaining a large scope of opinion from differ-ent groups of people situated in various locations.It is recommended in the literature that it may benecessary to spend a little more and engage in aninitial investment of time in this kind of meeting in order to increase the chances of participation.This in turn will increase the validity and generalis-ability of a study and fulfil the criterion of accept-ability to respondents. Although most of thestudies identified involved asking groups of health professionals, we conclude that the Delphitechnique is sufficiently reliable and valid to beused more widely to gain information frompatients and from the general public.

With regard to focus groups, there is ambiguityconcerning the ways in which focus groups should be conducted and whether this is a cost-effective way of acquiring opinion. The morerecent literature suggests that each study should be considered individually and that a set of rulescan work against this method because it can berestrictive. There is, however, consensus concern-ing their pitfalls. They do not use large samples,and are therefore not generalisable to widerpopulations, although this is not always the aim.Sometimes focus groups seek the opinions ofspecific sections of the community to get a feel forthe types of issues they consider to be importantand there is no desire to project these ideas tolarge populations. They are susceptible to selectionbias, in that only those with very strong feelings on an issue will participate, and in addition to that the most confident people will be heardwhereas the quieter members of the group mayfeel intimidated. Group composition can affect the data gathered. Careful consideration must begiven to whether it would be an advantage forparticipants to know one another, or to becomplete strangers; if people feel uncomfortablethen they may withhold information. Biases canalso occur through the facilitator of the discus-sions. Objectivity and skill of the researcher isessential in order to maximise reliability. It hasbeen shown that some sensitive topics are accept-able for group discussion; however, participantsmust feel comfortable with talking about thesetypes of issues with both the other members of the group and the facilitator. In this way previouslyuntapped information can be uncovered. If partic-ipants feel uneasy about disclosing personal infor-mation, they may withhold it and therefore affectthe validity of the study.

Citizens’ juries are a relatively new technique foreliciting preferences. Their major strength is thatpeople have the chance to deliberate over somevery complicated and involved topics. Participantsare provided with information from a number ofprofessional and lay people and can discuss withother jury members their beliefs and personalconclusions. For this method to succeed each jurymust be carefully conducted. It is a valuable tool if moderators are carefully selected and are highlytrained and experienced, and if a representativecross-section of the population is considered. Ifthese basic recommendations are not adhered to,then the jury may fail to reach conclusions onthese difficult and complex issues. It is also veryimportant to remember the criticisms presented by Price, namely that citizens’ juries are morallyand democratically irrelevant because they are not representative.

From the evidence presented it would appear that consensus panels are not an appropriatemethod of involving the public in priority setting.However, it may be necessary to conduct furtherstudies using this tool before it can be dismissed. It is less costly than a citizens’ jury and could be avaluable technique if panel members are carefullyselected and given adequate time for deliberation.A crucial issue not yet addressed, however, iswhether the questions being asked are those that lay people would ask.

Public meetings are commonly used and are aquick and cheap way of gaining opinion, but the disadvantages appear to far outweigh theseadvantages. They can be unrepresentative in thatthey only capture the opinions of those with aspecial interest and those willing to speak out.Generally, the public has become sceptical aboutthe motives behind these meetings because theyhave been used to inform people of a decision that has already been made. Additionally, it hasbeen noted that they can be inaccessible to thepublic because of failure to advertise and poororganisation. Despite this, it is recommended here that they should not be discounted untilthere has been further evaluation of what they add when used with caution and alongside other methods.

Complaints procedures are noted but these will only consider particularly negative view points, and have limited value in assessing public opinion. Further work is needed to establish the appropriateness of nominal group techniques, concept mapping and case study analysis.


57

So far this review has identified potentiallyuseful techniques for eliciting public prefer-

ences in the delivery of healthcare. Such anexercise is only useful if those views are taken into account, or at least deemed important, in the decision-making process. This chapter reportsthe methods and results of a pilot study set up toaddress this question. Although both qualitativeand quantitative techniques have been explored in this report, this chapter adopts a mainly quanti-tative approach to establishing the weight attachedto public preferences vis-à-vis other criteria used to set priorities. The next section presents themethodology of the study. The study was dividedinto two stages: a questionnaire-based surveyincluding choice-based CA* and an allocation of points exercise to establish the weights of thedifferent criteria†, and a telephone interview/follow-up questionnaire to further explore viewsconcerning involving consumers in decision-making, as well as views concerning the differenttechniques for eliciting weights. The results arethen presented and discussed.

Methods

Questionnaire-based surveyThe literature review described in chapter 2 led tothe identification of literature on priority setting.This literature was read with a view to identifyingcriteria, other than consumer views, which hadbeen used in priority setting. When reading thisliterature consideration was also given to frame-works for priority setting. The results from thisreview are shown in appendix 5. Following this,

discussions took place between one of theresearchers (AB) and health professionals involvedin priority setting to identify a set of criteria usedfor priority setting. The criteria included in thestudy are presented in Table 1.

In this chapter the focus is on the communityvalues and priorities criterion, and to what extent evidence citing the views of the communityis incorporated into priority-setting decisions.Three additional criteria – the severity of thedisease/condition, how many people it affects, and whether national/local priorities are fulfilled – were incorporated into the questionnaire byincluding them in the scenarios presented. Twoscenarios were presented: the first involved ahealth service for gynaecological cancer and thesecond a health service for asthma. This design was based on the a priori hypothesis that differentpreferences may exist under the different con-ditions. A copy of the questionnaire is given inappendix 6.

Choice-based CAWhen using choice-based CA, levels must beassigned to the criteria. These were decided with reference to the literature review anddiscussions with those involved in priority setting (see Table 1).

The criteria and levels defined in Table 1 gave rise to 9375 (55 × 31) possible combinations of healthcare proposals. Experimental designtables586 identified an orthogonal design of 25 proposals, which were paired into 13 choices.Given that two different scenarios were presented,

Chapter 7

The importance of public views in priority setting:the perspective of the policy-maker

* Also known in the literature as CA, stated preference and stated preference discrete choice modelling, and discrete choice experiments.† These two approaches make different assumptions about how individuals form their preferences. In the first, it is assumed that whilst respondents can provide an overall evaluation of whatever is being valued, they do not havethe ability to link the contribution of the individual components (or weights of these components) to the overallevaluation. Assuming this approach, the researcher must infer the weights that are implicit in the respondents’evaluations. In contrast, the second approach assumes that respondents know the individual weights they assign to the criteria or characteristics of the good being valued. Whilst there has been a large amount of psychological workcomparing these two approaches to decision-making,562 no such work exists in healthcare. This study compared the two approaches.

The importance of public views in priority setting: the perspective of the policy-maker

58

this resulted in 13 choices for each scenario. These 26 scenarios formed the basis of twoseparate questionnaires (Type 1 and Type 2)consisting of 13 discrete choices. The Type 1questionnaire included six choices from the firstscenario and seven from the second, whereas theType 2 questionnaire comprised the remainingseven choices from the first scenario and theremaining six from the second scenario.

Two dominant choices were included in eachquestionnaire to assess the internal consistency

of responses (definition in chapter 4). Respondentswere dropped from the analysis if they ‘failed’ bothtests on the basis that if they failed one test this maybe because of random error whereas if they failedboth tests this is more likely to represent a lack ofunderstanding of the process.

From the choice-based responses, the followingbenefit equation was estimated:

∆V = α 1HEALT + α 2CLIN + α 3BUD + α 4EQUIT+ α 5QUAL + α 6COM + e

where ∆V is the change in benefit (or utility) inmoving from proposal A to proposal B, and theindependent variables are the differences in thelevels of the criteria, as defined in Table 1. The α’sare the coefficients of the model to be estimated,indicating the marginal importance of a unitchange in the given criterion, and e is theunobservable error term‡.

The above equation was used to estimate theweights of the different criteria: α 6 is the weightassociated with the public’s views criterion and αn

the weights of all other criteria (n = 1, 2, 3, 4, 5).These weights indicate the marginal change inoverall utility, V, resulting from a marginal changein this given criterion. This marginal change isobviously dependent on the unit of measurement.So, for example, whilst the coefficient on potentialhealth gain indicates the marginal change in benefitof moving from say ‘temporary improvement now’to ‘sustained improvement later’, the coefficient on‘community values and priorities’ indicates the mar-ginal change in benefit in moving from say ‘weakevidence “object”’ to ‘robust evidence “indifferent”’.The Wald statistic was used to test whether thecriteria weights were significantly different using the following null and alternative hypotheses:

H 0: α 6 – α n = 0

HA: α 6 – α n ≠ 0

A general-to-specific approach was adopted. Thegeneral model includes all the criteria, whether or not they are significant. The most significantvariable is then dropped and the model re-run,repeating the same process until a model with only significant attributes at the 5% level remains.This latter model is referred to as the specificmodel. The Chow-type likelihood ratio test was

TABLE 1 Criteria defined with their corresponding levels

Criteria Levels for discrete choice experiment

Potential health • Life-saving nowgain (HEALT) • Sustained improvement now

• Sustained improvement later• Temporary improvement now• Temporary improvement later

Evidence of clinical • Empirical evidence effectiveness (CLIN) (MA, RCT, descriptive)

• Expert opinion• None

Budgetary impact • Big save(BUD) • Small save

• None• Small expense• Big expense

Equity of access • Big reduction in inequalityand health status • Small reduction in inequalityinequalities (EQUIT) • Remains the same

• Small increase in inequality• Big increase in inequality

Quality of service • Two or more direct ‘hits’(QUAL)a • One direct ‘hit’

• Two or more partial ‘hits’• One partial ‘hit’• No ‘hits’

Community values • Robust evidence ‘support’and priorities • Weak evidence ‘support’(COM) • Robust evidence ‘indifferent’

• Weak evidence ‘object’• Robust evidence ‘object’

a See the questionnaire in appendix 6 for definition of a ‘hit’in terms of the quality of the service

MA, meta-analysis

‡ Given the nature of the experimental design employed, interaction terms could not be considered.


59

used to investigate whether the weights for the different criteria differed according to thescenarios presented to respondents (gynaeco-logical cancer or asthma). If there was no differ-ence then the data sets could be merged andjointly analysed.

Allocation of pointsIn this study, points were used to define thebudget. Respondents were asked to allocate a totalof 100 points (deemed a manageable amount563)between the criteria identified. In order to encour-age honest preference revelation and to explicitlyincorporate strength of preference into thisexercise, respondents were reminded that thepoints they allocated to the different criteriashould total 100 and represent their strength ofpreference. An example of this type of question isillustrated in appendix 6. In the analysis the meannumber of points allocated to each of the criteriaacross all individuals represented the weight andhence the importance of that attribute. Analysisusing the median was also included and the results compared. Paired t tests were used to testfor statistical differences across the two scenarios. If there was no difference, the data sets could bemerged and jointly analysed. Using similar hypo-theses to those illustrated in the discrete choicessection, paired t tests were also used to testwhether the weights placed on the criteria weresignificantly different compared with the com-munity values criterion (at the 5% level). TheFriedman test was used to test whether there were significant differences between the way the criteria were ranked across respondents.

Sample and settingThe questionnaire was sent to 109 health policy-makers and healthcare professionals in Scotland:46 were employed at the health board level; 24 at healthcare trust level; and 39 comprised a convenience sample of University of AberdeenHealth Economic Correspondence Course students (the typical background of such students includes managers and providers of healthcare).

Telephone interviews and follow-upquestionnaireRespondents to the questionnaire were asked ifthey would be willing to be contacted to discusstheir responses. Those who agreed were initiallysent a letter reminding them of the study with acopy of their completed questionnaire. They wereinformed that they would be contacted by tele-phone. A structured telephone interview schedulewas used (appendix 7). The convenient sample ofcorrespondence course students completed theinterview schedule as a follow-up questionnaire.The aim of this further questionnaire was todirectly question respondents on the importanceof public views in the priority-setting process andto assess their views on the two techniques used to elicit criteria weights. Respondents were alsoasked to rank the criteria so that their responsescould be compared with the results from thequestionnaire-based survey.

Results

Questionnaire-based surveyOf the 109 questionnaires sent out, 57 werereturned in the required 6 weeks (with noreminders owing to the timescale). Of these, five were returned uncompleted. The remaining52 respondents all passed the dominant tests andwere included in further analysis, giving a usableresponse rate of 48%. Bias due to non-responsemay be a problem here given the relatively lowresponse rate. This could be investigated in alarger study by comparing the characteristics ofrespondents and non-respondents and analysingwhether the samples were statistically differentfrom one another. In this pilot, specific character-istics were not available for non-responders.

Table 2 shows the professional background of the respondents.

Choice-based CAThe Chow-type likelihood ratio test did not reject the null hypothesis of homogeneity; that

TABLE 2 Profession of respondents to the questionnaire

Profession Frequency

Health professional employed at healthcare trust level (e.g. associate medical director, clinical 13 (25%)GP coordinator, head of service)

Health professional employed at health board level (e.g. PHC, HE) 19 (37%)

Health Economics Correspondence Course students 20 (38%)

PHC, public health consultant; HE, health economist


60

is, the hypothesis that the coefficient results are the same irrespective of the scenario underwhich they were presented. The results gainedunder the two different scenarios could thereforebe merged and presented as one. The results from the regression equation are shown in Table 3.

The results from the specific approach show that, with the exception of quality of service, allcriteria had a significant effect on the choice of the health proposal. All criteria had a positiveeffect, indicating that the better the level of theattribute, the more likely the respondent wouldchoose that healthcare proposal. The weights of the criteria are given by the coefficients. A highercoefficient indicates that a unit change in thatcriterion had a higher effect on benefit. A unitchange in evidence of clinical effectiveness wasconsidered to be the most important of the criteria, followed by budgetary impact, communityvalues and then potential health gain. However,there was no significant difference betweenthe importance of community values and the other four criteria.

Allocation of pointsPaired t tests indicated no difference in theallocation of points according to the two different

scenarios. The results presented in Table 4 aretherefore for the combined data set.

Different marginal weights to those found usingchoice-based CA were obtained, with potentialhealth gain and evidence of clinical effectivenesscarrying the highest weights. Community valueswere the least important criterion. Paired sample t tests revealed significant differences (at the 5%level) between each of the criteria when comparedwith community values. The Friedman test indi-cated that there were no significant differencesbetween criteria ranking across respondents; thatis, health gain was ranked first (with the highestnumber of points) and community values last (with the lowest number of points) a significantnumber of times. Using median values gave similarresults to those found using the mean values withthe exception that both community values andquality of service were deemed equally to be theleast important criteria.

Telephone interviews and follow-upquestionnaireFrom the returned questionnaires, 15 respondentswere willing to be contacted at a later date todiscuss their responses in a telephone interviewand in addition 11 Health Economics Corres-pondence Course students completed a follow-up

TABLE 3 Results from choice-based CA

General Specific

Criteria Coefficient (weight)

Potential health gain (α 1) 0.3992** 0.3067**

Evidence of clinical effectiveness (α 2) 0.4773** 0.4756**

Budgetary impact (α 3) 0.4188** 0.3265**

Equity of access and health status inequalities (α 4) 0.3807** 0.2877**

Quality of service (α 5) 0.1206 –

Community values and priorities (α 6) 0.4082** 0.3202**

Number of observations 671 671

Log-likelihood function –294.9988 –297.3248

ACIa 0.897 0.901

Restriction/null hypothesis Chi-squared (p-value)

H0: α 1 – α 6 = 0 0.14 (0.71)

H0: α 2 – α 6 = 0 3.51 (0.06)

H0: α 3 – α 6 = 0 0.04 (0.85)

H0: α 4 – α 6 = 0 0.42 (0.51)

a Akaike’s information criterion (AIC) is a measure of goodness-of-fit. It is estimated as: AIC = –2InL(Mβ) + 2P/N, where L(Mβ) =log-likelihood of the model, P = the number of parameters in the model and N = the number of observations** p < 0.01


61

questionnaire of the same structure. However,given time constraints, six people were contactedby telephone: two were healthcare professionals attrust level and four were healthcare professionalsat board level. All 11 questionnaires from thestudents were used. Despite the small numbersinvolved, a number of interesting findingsemerged from the follow-up interviews and thequestionnaires (Table 5). Foremost, respondentsfelt that the important criteria in priority settinghad been covered in the first questionnaire.

Regarding the choice-based CA approach toeliciting weights, 11 of the 17 respondents thoughtthat this was a realistic or very realistic approach.Ten respondents found such choices very difficultor difficult. However, in retrospect, the term ‘easeof making choices’ does not distinguish betweenwhether the questionnaire was difficult to under-stand or whether the questions were difficult toanswer (i.e. whether the answers required carefulconsideration). The two are quite different.Although we may be looking for ease of compre-hension, when we are considering a difficultsubject such as priority setting, we would notexpect the choices to be easy. This conclusion is supported by the fact that studies applying the technique to the general public and patientshave reported respondents having little difficulty

completing the choices.191,194,199,201,202,203,396 Choice-based CA assumes that individuals consider all the criteria in the study, and that they are willing to trade between these. However, nine of the 17 respondents indicated that they focused in on some of the criteria. This suggests that futurework should explore in more detail the decisionheuristics that respondents employ whencompleting choice-based experiments.

Six of the 17 respondents thought that theallocation of points exercise was difficult or verydifficult. Again, this may reflect the nature of thequestions being asked. In the allocation of pointstask, individuals were encouraged to think abouttheir strength of preference. However, the resultssuggest that this may not have been the case, andthat the resulting weights may be ordinal.

Despite community values and priorities beingconsidered the least important criterion in theranking exercise (Table 6), 15 of the 17 respon-dents thought that the community had a role toplay in priority setting and 12 indicated that thisrole was important or very important.

A summary of rankings obtained from the results of the different techniques is shown in Table 7.

TABLE 4 Results from allocation of points exercise

Criteria Minimum Maximum Mean Relative Friedman Median Relativenumber number number importance test mean number importanceof points of points of points rank of points

Potential health gain 15.00 65.00 28.5 1 5.44 27.00 1

Evidence of clinical 5.00 60.00 23.1 2 4.58 21.75 2effectiveness

Budgetary impact 0.00 30.00 15.7 3 3.52 16.83 3

Equity of access 0.00 30.00 13.3 4 3.04 15.00 4and health status inequalities

Quality of service 0.00 20.00 11.0 5 2.53 10.00 5

Community values 0.00 20.00 8.5 6 1.89 10.00 5and priorities

Paired sample t tests t test p-value

Potential health gain – community values 11.063 0.000

Evidence of clinical effectiveness – community values 8.474 0.000

Budgetary impact – community values 6.032 0.000

Equity of access – community values 5.373 0.000

Quality of service – community values 2.901 0.005


62

TABLE 5 Responses to the telephone interview/follow-up questionnaire

Question Responses

Were any criteria irrelevant? Yes No1 13

Discrete choicesVery hypothetical Hypothetical Realistic Very realistic

Realism of choices posed? 1 4 6 5

Very difficult Difficult Easy Very easy

Ease of making choices? 1 9 4 2

Consider all the criteria Focus on some criteria

Decision heuristics? 8 9

The allocation of pointsVery difficult Difficult Easy Very easy

How easy did you find 0 6 8 2allocating the points?

Yes No

Reflection of strength 12 5of preference?

Comparing the methodsThe paired choice exercises The allocation of Neither

points exercises

Which of the two methods 5 6 4did you find easiest?

Which of the two methods 3 7 5did you find quickest?

Priority setting and the communityYes No

Do you think that the community 15 2has a role to play in priority setting?

Of no importance Of little Important Very importantimportance

How would you rate community 0 4 4 8views when deciding whether or not to implement a proposal?

TABLE 6 Criteria ranked in order of importance by telephone interview/follow-up questionnaire

Criteria Mean ranking Rank

Potential health gain 1.1 1

Evidence of clinical effectiveness 2.8 2

Equity of access and health status inequalities 3.4 3

Quality of service 3.7 4

Budgetary impact 4.8 5

Community values and priorities 5.3 6


63


A choice-based CA approach was used to estimateweights indirectly, and the allocation of pointsapproach was used to estimate weights directly.Both approaches found the views of the public to be important in the priority-setting exercise,although the relative rankings differed across thetwo techniques. The results from the allocation of points exercise were similar to the directranking exercise carried out in the telephoneinterview/follow-up questionnaire. Fundamentally,the conclusion to be drawn from these results isthat the weights assigned to the various criteriaappear to differ depending on the elicitationmethod used. Following from this, the context in which the technique is to be used should betaken into account.

This may be an obvious, necessary conclusion for some. Cookson points out that a number of psychologists and behavioural theorists haveargued that preferences are constructed inresponse to stimuli rather than revealed ordiscovered.587 This implies that if preferences are constructed in different ways (given thedifferent methods), different preferences will be elicited, implying predictably different results.In this case, the different assumptions maderegarding the CA approach, the allocation ofpoints method and the direct rankings may bereflected in the results. Expanding this further,Clark notes that eliciting precise results from the budget pie technique (termed allocation ofpoints here) is dependent on the fulfilment ofsome basic assumptions.277 These include issuessurrounding the use of a monetary budget, theelicitation of honest preferences and the revelation of intensity of preference.

The problems encountered with the use of moneyin the allocation of points method were avoided byusing a points system. The main focus of discussionis therefore honest preference elicitation broadlyand, intertwined with this, the elicitation of theintensity of preferences. First, honest preferenceelicitation is not a problem specific to the allo-cation of points exercise; however, within thismethod there is room for strategic games to takeplace. This is likely to happen if respondents to the exercise are highly informed about thesubject area and the subject matter is of interestand highly important.277 In this study, it is assumedthat respondents will report honest preferences.The results from the exercise reveal little variationbetween the mean and median criteria valuesindicating, to some extent, that there were fewoutlying strategic values and most people reportedhonest preferences.277 This is not particularly sur-prising because it is not clear, in this case, whatbenefit would ensue from strategic behaviour.However, as a method whereby the weights areelicited directly, it is susceptible to such behaviour.In contrast, it may be argued that the implicitelicitation methods of the choice-based CAmethodology would incite low strategic behaviour.However, even when a robust design is used thereis still a concern by some authors who believe thatrational individuals will never truly reveal theirpreferences.282

Secondly, it is debatable whether the allocation of points methodology can successfully incorporateor take account of the intensity of respondentspreferences. The telephone interview/follow-upquestionnaire revealed that a number of respon-dents (5/17) did not allocate points in a way thatreflected their strength of preference. Elsewhere,Hoinville and Courtenay carried out an exercise

TABLE 7 Summary of the rankings obtained from the results of the different techniquesa

Criteria Choice-based CA Allocation of points exercise Direct ranking exerciseb

Potential health gain 4 1 1

Evidence of clinical effectiveness 1 2 2

Budgetary impact 2 3 5

Equity of access and health status 5 4 3inequalities

Quality of service – 5 4

Community values and priorities 3 6 6

a Whilst the rankings for the choice-based CA represent the impact of a marginal change in the criteria, the allocation of points anddirect ranking results are for the criterion generallyb Telephone interview/follow-up questionnaire


64

where the number of points they used was suc-cessively reduced and they revealed that respon-dents’ allocations to the various options were not reduced proportionately.564 Although there are other issues that cloud the reasoning on why this should happen, such as repeating theexperiment and the optimal number of points that should be used, one interpretation is thatrespondents’ strengths of preference are notindicated using this method.

However, it is not clear in this instance whether the differences in the results of the two methodsare due to a failure of the two techniques toproduce converging results (as discussed above,the allocation of point methodology may fail totake into account strength of preference or itsexplicit nature may be open to manipulation), orthe fact that the techniques themselves elicit truepreferences but they are different because thepreferences are constructed in different ways.Other factors that may have influenced the resultsare design issues specific to this study rather thanthe techniques per se. The study focuses mainly onqualitative criteria which, even though given anordered description, raises questions about theconsistency of interpretation across individualsthrough the nature of the attribute levels. In theallocation of points exercise, respondents werepresented with a list, as in shown in appendix 6;ranking the criteria in this way can influence (bias) respondents’ ranking. Additionally, a listformat causes problems for respondents having to divide points between so many criteria (six), and trying to think simultaneously of how manyextra points one criterion deserves over anothercompared with all others may be unnecessarilycomplicated and alter the focus of the exercise.Respondents in this case were generally motivatedto use simple numerical amounts that wouldemphasise a preference of one criterion overanother but not necessarily indicate strength of preference or maintain it when compared with the other criteria. An exercise that involvesthe division of a pie chart into ‘slices’ repre-senting criteria importance may overcome these problems.

In conclusion, it is recommended that furtherwork should seek to develop these methodologies.Future work should also aim to incorporate themethodologies into the priority-setting process. In the UK (and, in fact, elsewhere) various frame-

works are used to elicit and incorporate criteriaweights and scores into priority-setting decision-making (see appendix 5). The examples, identi-fied from the literature review, highlight threeframeworks (or weighting and scoring methods) –rating exercises, allocation of points techniquesand discrete choice experiments – that arecommonly used for the purposes of scoringpriorities. The first of these, assigning weights on a scale from 1 to n, has a number of limitations.First, there is no recognition of the need to maketrade-offs between the criteria. Given this, it ispossible that such a weighting exercise will resultin high weightings being assigned to all criteria.Secondly, it is not clear whether such an approachindicates strength of preference (i.e. if onedimension is assigned a weight of 1 and another 5, does this mean that the latter is five times asimportant as the former)? Also, it has been shownthat asking individuals to value dimensions ofbenefit individually will lead to different results to those arising from establishing weightings forthe same dimensions when they are defined as a package.475,531,588,589,590

Ideally, a technique for weighting criteria shouldindicate not just the order of the weighting, butalso the strength of preference for each criterion.Given that no one method is currently used inisolation or to its full potential, it is proposed thatthese limitations may be overcome by the adoptionof either a discrete choice experiment or theallocation of points method§, and future workshould explore this.

Further, this study investigated the importance ofcommunity views in priority setting from the per-spective of the policy-maker. However, an equallyimportant perspective is that of the consumer andwhether they want to be involved in priority-settingdecisions (see chapter 8). The approach adoptedin this chapter could easily be adapted to elicitpublic views concerning their involvement indecision-making. Experience regarding the elicit-ing of CHC views (another pilot study using thisquestionnaire indicated that CHC members haddifficulty interpreting the meaning of the criteria)would suggest that alterations would be required tomake it more understandable. Although the publicdoes have the ability to answer choice-based CAquestions if framed correctly (see chapter 5 forreferences applying to this technique), the publicperspective should be a future area of research.

§ See Farrar and colleagues211 for an example of applying discrete choice experiments within a priority-settingframework.


65

Results from the systematic reviewQuantitative techniquesQuantitative techniques, classified as ranking,rating or choice-based approaches, were evaluatedaccording to eight criteria: validity; reproducibility;internal consistency; acceptability to respondents;cost (financial and administrative); theoreticalbasis; whether the technique offered a constrainedchoice; and whether the technique provided astrength of preference measure.

Ranking exercises included simple rankingquestions, QDP and CA. Simple ranking exerciseshave proved popular, probably because of the easeof both devising a ranking exercise and analysingthe resulting data. However, the results of suchexercises are of limited use. The QDP has not been used to date in healthcare. Given its ability to deal with the vagueness that exists in humandecision-making, it may be particularly attractive in healthcare, where preferences are known to be both vague and difficult to articulate. CA ranking exercises provide useful information to policy-makers, and did well against the above criteria.

A number of rating scales were identified. The VAS has proved popular within the QALYparadigm. The main limitations of this techniqueare the lack of any constrained choice and thedoubts expressed over whether the techniquemeasures strength of preference. CA rating scalesdid well against the above criteria. A number oftechniques were identified for eliciting attitudes,including Likert scales, the SDT and the Guttmanscale. Satisfaction surveys have been frequentlyused to elicit public opinion. Their popularityprobably reflects the relative ease of carrying out such surveys. Researchers should ensure that they construct sensitive techniques, or else use generic techniques where validity hasalready been established. Even when well-designed, sensitive techniques are used, usersshould be aware of the limited use of suchtechniques at the policy level. SERVQUAL appears to be a potentially useful technique and future research should consider its application in healthcare.

A number of choice-based techniques wereidentified. Three such techniques – MoV, AHP and allocation of points technique – have hadlimited application in healthcare, resulting in asmall literature on their methodological status.Those more widely used in healthcare – SG, TTO, discrete choice CA and WTP – did wellagainst the above criteria. Little methodologicalwork is currently available on the PTO.

Qualitative techniquesWhilst chapter 5 defined a set of criteria forevaluating qualitative techniques, very few of thetechniques identified for the review have beenassessed according to these criteria. Althoughmethodological issues can be extracted from the studies they are not explicitly addressed, and extracting them systematically has beenchallenging. Nonetheless, we believe that it was a useful process. We also conclude that it is necessary to continue the attempt to drawanalogies between evaluations of quantitative and qualitative methods because users of thetechniques should be able to choose eitherapproach as best fits their purpose and context. It is also necessary to assess the several checklistsset out in chapter 5 in respect of their ability tocapture the criteria that we have used here.

In this review we attempted to assess qualitativetechniques according to six criteria: validity;reliability; generalisability; objectivity; acceptabilityto respondents; and cost. All the methods werefound to have distinct strengths and weaknesses,but there is a lot of ambiguity in the literature.There is disagreement about whether it is advan-tageous to use individual or group methods, butthis very much depends on the specific topic being discussed and the people being asked. The most popular and widely used methods were one-to-one interviews and focus groups. In both of these methods it is crucial that theinterviewer/moderator remains as objective aspossible. It is impossible for the researcher to be completely objective but steps must be taken to minimise the opportunity for researcher bias.Both of these methods have potential problemswith validity and reliability. It is the responsibility of the researcher to minimise these problems at all stages of data collection, analysis and

Chapter 8



66

presentation of results. There were also manyapplications using the Delphi technique. In mostof the examples listed experts or professionalsparticipated but in some, however, patients wereasked to take part; it is proposed that the Delphitechnique could be more widely used to gainpatients opinion. Citizens’ juries were found to be very useful, especially when the subject matter is very complex. It is argued in the literature that as the participants have the opportunity fordeliberation, their final decisions and opinions are more valid and reliable than an uninformedreaction to a question. There are problems withgeneralisability, however, because only very smallnumbers of people can be involved and it is verytime-consuming and therefore costly. Consensuspanels were found to have similar methodologicalconsiderations to citizens’ juries but have beencriticised for not allowing sufficient time forparticipants to make decisions. They are, however,less costly and cannot be dismissed at this stage.Public meetings are frequently used and are aquick and inexpensive way of gaining publicopinion. It has been argued, however, that they are unrepresentative and therefore have limitedvalue. Complaints procedures are noted but willonly consider particularly negative view points, and have limited value in assessing public opinion.Three additional methods are included: nominalgroup technique; concept mapping; and case study analysis. It is proposed that they may beuseful, but further research is needed to test their use for issues in priority setting.

Results from primary research

Both the choice-based CA technique and theallocation of points method found the views of the public to be important in the priority-settingexercise, although the relative rankings differedacross the two techniques. In the follow-up tele-phone interviews, whilst the majority of respon-dents stated that the community had a role to play in decision-making, and that this role wasimportant or very important in the context ofpriority setting, they ranked this criterion as the least important of the six.

General issues raisedAlthough not the main concern of this study, theliterature searches highlighted a number of moregeneral issues that will be raised as attempts aremade to elicit public preferences for use in thepriority-setting exercise. These are independent of the techniques employed and include the following.

Who are the public?An important question within the context ofeliciting public preferences for use in prioritysetting is whose values should be used? Should weobtain values from actual users of the service orshould we elicit the views of the community moregenerally? There is no agreed answer to this ques-tion. Gafni591 argued that, within the context of apublicly provided healthcare system, it is the viewsof the community that are relevant. Shackley andRyan,396 however, argue that the answer to thisquestion depends on the context of the questionbeing addressed. If policy is concerned with whathealth services should be provided (hearts versuships versus helicopters) then it is the views of thecommunity that are relevant. However, if questionsare concerned with how to provide, then thepreferences of patients are relevant.

Do the public want to be involved in healthcaredecision-making?An implicit assumption so far has been that thepublic want their views to be considered. In arecent review of published studies, Benbassat andcolleagues592 concluded that whilst most patientswant to be informed about their illness, only aproportion want to be involved in planning theircare, and that some patients would prefer to becompletely passive. Determinants of a desire to be passive include medical condition and socio-economics factors. Similar conclusions werereached by Guadagnoli and Ward.593

In a survey conducted by the British MedicalAssociation and The King’s Fund,594 only 22% of the general public thought that they shouldmake prioritisation decisions. However, in thisstudy the level of decision-making was restricted to prioritising amongst individuals. In contrast, in a large study conducted by Bowling,595 based on a random sample of the British population, it was found that most respondents (88%) thoughtthat surveys of public opinion should be used inplanning services. Groves,17 in the context of theBritish Medical Association survey, found that one-third of the public sample thought the public should have a say in the process. Thiscompared to half the managers and one-fifth ofthe doctors. In the qualitative literature, Coote and Lenaghan331 reported on a series of pilotcitizens’ juries. One of the juries addressed thequestion of whether the public should be involvedin priority setting. This jury concluded that thepublic should be involved in decision-making inconjunction with other experts. Across all fivecitizens’ juries, jurors noted that the importance of public involvement was the main motivational


67

reason for their involvement. The authors con-cluded that jurors were willing to be involved inthe decision-making process, although financialremuneration may be necessary to compensate for their time, and it should be remembered that juries are to compose only one part of thedecision-making process. Likewise, Lenaghan and colleagues332 observed that given “enough time and information” the public is “willing andable” to contribute to the priority-setting process.Kneeshaw,596 reviewing public involvement inhealthcare, reported that the public wanted to be involved in the decision-making process,although the final decisions should be left todoctors. Related to this, there is also someevidence that there is disutility associated with patient involvement in the decision-making process.597,598

Do public preferences for healthcare exist?Researchers concerned with eliciting publicpreferences in the delivery of healthcare make animplicit assumption that such preferences exist.Within psychology, Fischoff599 made a distinctionbetween the philosophy of articulated values andthe philosophy of basic values. The former repre-sents a state where individuals have well-articulatedvalues, which the researcher can extract. The latter holds that individuals lack well-formedpreferences for all but the most familiar of goodsor services. This familiarity is a result of havingmade numerous choices in the past (and errors),such that a complete preference structure can besettled on. In reality, individuals’ preferences forany given good or service may lie somewhere on a continuum between these two philosophies.Psychologists have also developed the notion of ‘constructive nature of preferences’ to explain preference reversals in experimentalstudies.478,600,601 Here it is argued that individuals‘construct’ their preferences as they answerexperimental questions.

Given the nature of the commodity healthcare,preferences for this good may not be complete.Individuals often do not have experience ofhealthcare interventions. There is an asymmetry of information and agents often make decisions on behalf of patients and the public. There is also, in many publicly provided healthcare systems,a lack of choice*. Thus, healthcare may be betterplaced under a ‘basic values’ philosophy. Shiell

and colleagues602 noted the possibility of peoplenot having well-formed preferences for health and healthcare, arguing that the values elicitedfrom experimental studies may not equate withactual preferences. However, no empirical workwas carried out. Dolan and colleagues323 found that that respondents changed their viewsconcerning priority setting in healthcare betweentwo questionnaires, and argued that this change inpreferences was due to discussions held betweenthe questionnaires. (See page 68.)

Such findings have implications for the wayresearch is carried out into public preferencesconcerning healthcare. If respondents really doconstruct their preferences as they go along, then‘valid’ results may only be obtained after successivesurveys (quantitative or qualitative) or interactivediscussions have taken place.

Clarke567 advocates the use of deliberation infinding out public opinion. It is believed that ifpeople have the chance to deliberate then theirjudgements and conclusions will be more validthan mere gut reactions to a question. The deliber-ation process involves participants reasoning,reflecting and refining their ideas, all of which can lead to an informed and committed decision.There are various degrees of deliberation in themethods previously described in this review. Clarke states that focus group discussion involves a low level and citizens’ juries involve a high level,and that with increased levels of deliberationcomes increased cost.

Clarke567 proposes a number of methods to helpthe deliberation process. First, community issuegroups. Such groups lie between focus groups and citizens’ juries on the scale of deliberation.They consist of groups of 8–12 people who meetseveral times over a set period of time. Thesegroups then join other similar groups to form anetwork that is committed to discussing commonissues. Secondly, community workshops, whichhave a variable level of deliberation, are more in-depth than a focus group but can obtain a larger sample than a citizens’ jury. These consist of 12–20 people considered to be representative of the wider community. They meet for 1 day to discuss a particular issue with the aid of amoderator. Recommendations are made to acommissioning body. This method is subject

* Whilst people in a publicly provided healthcare system do not have much experience with choosing betweenhealthcare interventions and paying for them, such an argument may not apply in privately provided healthcaresystems. Thus, in such systems, preferences may be more complete and stable. This is ultimately an empirical issue.


68

to the same potential problems as some of theother techniques; that is, problems of gaining arepresentative sample and ensuring the use of askilled and unbiased moderator.

The Sedgefield Health Alliance formed a localadvisory group to help the deliberation process.603

It was set up between the County Durham HealthAuthority, the Sedgefield Borough Council andDurham County Social Services. The aim of joining these groups together was to improve the health and quality of life of people living andworking in the area. It was proposed that this beused as a method of enabling the exchange ofinformation and debate of relevant issues and toraise the profile of health and social issues and tostimulate local interest in these issues. A total of 19 meetings were held during the period April to June 1999. A variety of deliberative methodswere used during these meetings and a number of concerns and issues were identified by theparticipants. As a means of validation, the publicrequested a report summarising the topics raisedand asked that information should be fed back tothem at organised liaison meetings. The authorsstate that this method’s success lies in the con-viction of board members to appreciate andrespond to the issues raised in the meetings – an assessment of the outcome rather than of the process of consultation.

Quantitative and qualitative techniques may beused together to help the deliberation process.Dolan and colleagues323 conducted a series ofstructured focus groups to investigate peoples’views about priority setting and how these viewschange after discussions in the groups. Theauthors randomly selected 72 people from sixage–sex categories. Participants were split into ten groups and told they would be paid £30 forattending after the conclusion of the secondmeeting. The discussion groups were moderated by two of the researchers who defined the agendaand gave each group member the opportunity to speak. Two meetings with each group wereconducted, separated by 2 weeks. The partic-ipants were asked to fill out a questionnaire at the beginning of the first meeting and at the end of the second to determine how theirviews had altered after the opportunity fordeliberation. The questionnaire consisted of two groups of questions. First, participants wereasked whether certain groups (e.g. doctors andnurses, politicians) should have more or lessinvolvement in making decisions about priori-tising healthcare. Secondly, they were askedwhether a specified group of people (e.g.

children, disabled, married people) should have a higher or lower priority for treatment. Both ofthese questions used a four-point Likert scale.Participants were also, at the start of the secondmeeting, asked to prioritise between four people –all of whom needed treatment for differentreasons. The authors found that participants’ views changed after the two meetings to the extent that they no longer discriminated quite soheavily against smokers, drinkers and drug takers.This study shows that quantitative research can be cultivated from focus groups.

Preferences for the status quoWith regard to eliciting public preferencesregarding new health technologies, there is someevidence that people will adopt a conservativeresponse, preferring the status quo.61,439,604–606

Cartwright604 found that within the context ofmaternity care, women tend to choose the optionsthey have experienced. Similarly, Porter and Mac-Intyre,439 in a study concerned with innovations in antenatal care, found that pregnant womenpreferred the type of care they received. Ryan andcolleagues61 found that, within the context of aRCT, individuals who experienced the introductionof a patient health card valued it, whereas thosewho had no experience did not value it. Bate andRyan605 argued that preferences for junior doctor-led care over nurse-led care in the provision ofrheumatology services could be partly explained by a lack of experience with the latter.

This decision-making heuristic has importantimplications in a climate where public views arebeing promoted within the context of prioritysetting. The logical implication is that if the publicprefer what they know, new innovations may not be valued sufficiently to persuade them to change from the status quo.

Explanations for preference for the status quohas received attention in the economics literature.This anomaly in choice is known as the endowmenteffect607 (also termed status quo bias608) and refersto a situation whereby people value goods morehighly once they own them (or have experience ofthem). This results in individuals often demandingmuch more to give up an object than they wouldbe willing to pay to receive it. The endowmenteffect has been offered as an explanation for thewell-documented discrepancies between WTP andWTA (willingness to accept) in the contingentvaluation literature.609–613

Another hypothesis for such conservativepreferences is that individuals are attempting


69

to minimise the potential to experience thepsychological feelings of regret and disappoint-ment. In explaining consistent violations of EUT,Bell476,480 and Loomes and Sugden477,481 argue thatwhen making decisions, individuals take accountnot only of the final outcome but also of thechance of experiencing regret and disappoint-ment. As indicated above, evidence has suggestedthat regret is an important element in individualvaluation and decision-making in healthcare.482,483

The potential to experience both regret anddisappointment will be minimised if individualschoose the status quo.

Kahneman and Tversky argue that that lossaversion may help to explain violations of EUT.475

Here, a reference point, usually the status quo,determines that individual preferences and thedisutility of giving up an object is greater than theutility associated with acquiring it. The anomaliesof regret and the endowment effect both manifestthemselves within the concept of loss aversion.

Within the health service research literature, alimited amount of work has attempted to explainpreferences for the status quo. In attempting toexplain why pregnant women prefer the care they receive, Porter and MacIntyre439 argued thatpatients assume that whatever service is offered has been carefully considered by experts and istherefore likely to be the best for them. That is,‘what is, must be best’. Such beliefs naturally resultin aversion to innovations. Ryan and colleagues61

and Bate and Ryan605 both argued that thetendency to favour the status quo may be explainedby a lack of information about the alternative.Under such circumstances, the potential for status quo bias will be increased if individuals are risk averse.

Ethical issues in involving the publicAssuming that the public (however defined) want to be involved, that they have well-definedpreferences, and that techniques are used toovercome potential problems of preferences forthe status quo (as well as satisfying criteria for agood technique), is public involvement a goodthing? Involving the public in the planning andprioritising of services is fraught with ethical as well as practical issues. One major issue that will be encountered is how to deal with conflictingviews about priorities (both between subgroups ofthe population as well as the public and healthprofessionals). Potential differences in opinionimply that important ethical issues will be raised asattempts are made to incorporate public opinioninto priority setting. For example, Stronks and

colleagues,336 in a study using consensus panels,observed that patients and health insurers wereacting in their own interests. The same could notbe said of the public or healthcare professionals(see also Dicker and Armstrong289). The publicfavoured individual responsibility (i.e. co-payments) without any consideration of the groups this may exclude. This strategy maythreaten the concept of equal access for equalneed. In contrast, professionals took into account the consequences of their decisions on equality of access. Stronks and colleagues336

concluded that

“it is not clear that including all the different actors in the decision-making process of prioritisation ofhealth services will lead to more equitable or broadlysupported outcomes or to better health for thepopulation ... their decisions might threaten theuniversal accessibility of core services”.

Studies identified in our review found that thepublic have set views over who they would prefer to see treated, with females being preferred tomales, non-smokers to smokers, poor people torich, non-drinkers over drinkers, whether Britishborn, Christians over atheists.18,25,26,68 Issues willinevitably be raised here concerning the extent towhich policy-makers take such views into accountwhen allocating scarce resources. What if policy-makers do not like the views of the public?

Overcoming tokenismIt is important that the public believe that theirviews are going to be incorporated into thepriority-setting process, and do not view suchelicitation exercises as tokenism. ‘Planning for real’ has been proposed as a method to ensurethis.614 This method was originally devised to aidpeople in planning their physical environment and surroundings in a user-friendly fashion. Kleinhas suggested that the theory of this methodtransfers very well in a health context.614 It involvesconsulting the public at all levels related to issuessuch as who to involve, what level of involvement to seek, who to consult, what the task is and whattype of support is needed to achieve this task. This will ensure that the public take the exerciseseriously. Also advocated is the encouragement of people to play an active role in creating newideas, plans and projects. However, ‘Planning for real’ has not been evaluated.

Bringing together quantitative and qualitative techniquesMany applied health (service) researchers arguethat qualitative and quantitative methods can andshould complement each other. Pope and Mays157


70

highlighted three ways in which this could be done:

• First, many quantitative researchers are familiarwith the idea of using qualitative methods inorder to prepare their quantitative techniques;for example, using interviews and/or focusgroups to determine the questions for a postalquestionnaire, the content of an outcomeindicator, or the criteria to include in a CA study.

• Secondly, qualitative techniques can be used in parallel with quantitative ones, either (1) to help explain quantitative findings; (2) toenlarge on such findings; or (3) as part oftriangulation.615 Whilst quantitative techniques

tell us that the public prefer local clinics or donot want resources to be allocated to certainsocial groups, qualitative methods can help usunderstand the reasons behind such preferencestructures. Using both types of techniques inparallel can be part of a validation process, as in a triangulation exercise whereby thedifferent techniques address the same issue from a slightly different direction.

• Finally, qualitative methods may be used toexplore “complex phenomena or areas notamenable to quantitative research”.157 Anexample of this would be using qualitativetechniques to establish the cognitive strategiesand decision-making heuristics employed whenresponding to quantitative questionnaires.


71

Recent UK Government policy suggests anincreased role for the public in priority setting.

The purpose of this review has been to identify andevaluate techniques for eliciting public preferences,and to assess the importance of public preferencesvis-à-vis other criteria for setting priorities. Con-sideration was also given to general issues that wereidentified in the literature that will be raised aspublic preferences are elicited for use in prioritysetting. This final chapter includes recommend-ations on what techniques to use. Finally, recom-mendations for future research are proposed.

Guidance on the use of techniques

It is concluded that there is no single best methodto gain public opinion. There are no hard and fastrules on which method should be used to answer acertain question, but the method must be carefullychosen and rigorously carried out in order toaccommodate the question being asked. Authorsshould be encouraged to set out their critiquesmore systematically.

Quantitative techniquesBased on the evidence currently available, for the ranking approaches, we recommend thatresearchers consider the QDP and CA. From therating approaches, CA should be further explored.Satisfaction surveys may be a useful technique tomeasure outcomes and to assess consultations andpatterns of communication. Researchers usingsuch surveys should ensure that they constructsensitive techniques, or else use generic techniqueswhere validity has already been established. Evenwhen well-designed, sensitive techniques are used,users should be aware of the limited use of suchtechniques. SERVQUAL appears to be a potentiallyuseful technique and future research shouldconsider its application in healthcare. Likert scales,Guttman scales and the SDT all provide usefulinformation on attitudes. All the choice-basedtechniques should be (further) explored.

Qualitative methodsSeveral qualitative methods were identified whichshould be considered when eliciting public

preferences. The technique used will depend onthe question being asked and also the types ofpeople being asked. Individual interviews andfocus groups are particularly useful in gainingdetailed data from a limited number of individuals;these may be the best ways of looking into sensitivetopics. The Delphi process, thus far mainly used in eliciting responses from professionals, could bemore extensively used to gain the public/patientperspective. Citizens’ juries have been shown to beextremely useful in looking at complex issues butare very costly and require a lot of preparation andplanning. Public meetings are a quick and cheapmethod of gaining a feel for public opinion butmay only take very extreme views into consider-ation. Further work may need to be conducted totest the value of using consensus panels in thiscontext, and complaints procedures have limitedvalue because they only consider very negativeviews of healthcare. Additionally, further work maybe needed to establish the appropriateness ofusing the special analysis techniques of nominalgroup technique, concept mapping and case study analysis.

Future research

The research agenda is broken down into researchquestions relating to the systematic review oftechniques and those relating to more generalissues that were raised whilst conducting thereview. The research agenda related to the cog-nitive strategies and decision-making heuristics thatrespondents adopt when completing quantitativesurveys should be seen as a priority. The generalissues emerging should be given equal priority.

Researching techniques:• the techniques recommended above should

continue to be researched• research to investigate the AHP, MoV,

allocation of points, QDP, SERVQUAL and PTO as quantitative methods, and telephone,email and dyadic interview techniques, con-sensus panels, case study analyses, conceptmapping and nominal group techniques as qualitative methods

Chapter 9

Recommendations on the use of techniques and future research

Recommendations on the use of techniques and future research

72

• when addressing the above points, quantitativeand qualitative methods should be used morealongside each other. A crucial part of thisresearch agenda should address the extent towhich preferences for healthcare exist, as well as the cognitive strategies and decision-makingheuristics respondents adopt when completingquantitative surveys. This should involveextensive piloting work using qualitativeapproaches to inform the design andinterpretation of quantitative studies.

General issues raised in the review:• do the public want to be involved in healthcare

decision-making?• potential problems encountered with a

preference for the status quo• ethical issues in involving the public

• developments of frameworks to ensure publicpreferences are incorporated into priority setting.

The recommendation to apply qualitative research to look at the cognitive strategies anddecision-making heuristics individuals employwhen answering such questionnaires will haveimplications for whether the techniques should be used to elicit public preferences. Therefore, we would recommend that this review of quanti-tative instruments for eliciting preferences beupdated to take account of work done in this field over the coming years. It is difficult to predictat the moment how quickly this research will beconducted. Given the nature of such research, it is unlikely that a substantial body of literature will exist in the next couple of years, and such anupdate should be considered in 5–10 years time.


73

This study was commissioned by the NHS R&DExecutive’s Health Technology Assessment

programme, project number 96/49/04.

This work is the product of a collaboration amongeight authors. Moira Napper was responsible forthe identification of the techniques within thesystematic review. She was helped by David Scottfor the quantitative methods and Caroline Reevesfor the qualitative methods. Mandy Ryan andDavid Scott carried out the review of quantitativetechniques and Caroline Reeves the review ofqualitative techniques. Edwin van Teijlingenhelped with the review of the SDT, Guttman scalesand satisfaction surveys and Christian Robb withthe review of techniques for visual analogue, SGand TTO. Angela Bate and Mandy Ryan carriedout the primary research concerned with theimportance of public views in decision-making and these authors, together with David Scott,carried out the primary research looking at theweights of the methodological criteria. MandyRyan took overall responsibility for writing thereport. Edwin van Teijlingen and Elizabeth

Russell made invaluable comments to all aspects of the report, and were involved in thewriting up. The views expressed in this report are those of the authors, who are also responsible for any errors.

The authors would like to thank Penelope Mullen, Shirley McIver (both from the Universityof Birmingham) and Shelley Farrar (HealthEconomics Research Unit, University of Aberdeen)for their comments on earlier drafts. Thanks also go to Anne Bews, Flora Buthlay, MargaretBeveridge and Sophie Davidson for their help with the photocopying and mailing of thequestionnaires. The authors are also indebted to all the respondents to the questionnaire-basedsurveys and to Andrew Walker (Greater GlasgowHealth Board) for his help with the primaryresearch assessing the importance of publicpreferences in priority setting. Finally, thanks to the anonymous referees who providedinvaluable comments and HTA series editors for their editing, proof-reading and, above all, perseverance.

Acknowledgements


75

1. NHS Management Executive. Local voices: theviews of local people in purchasing for health.London: NHS Management Executive; 1992.

2. Working for patients. Command Paper 555.London: HMSO; 1989.

3. Promoting better health. Command Paper 249.London: HMSO; 1989.

4. The health of the nation. Command Paper 1986.London: HMSO; 1991.

5. NHS Centre for Reviews & Dissemination.Undertaking systematic reviews of research oneffectiveness: CRD guidelines for those carrying out or commissioning reviews. York: NHS CRD,University of York; 1996. NHSCRD Report no. 4.

6. Black N, Brazier J, Fitzpatrick R, Reeves B, editors.Health services research methods: a guide to bestpractice. London: BMJ Books; 1998.

7. Bryman A. Quantity and quality in social research.London: Routledge; 1993.

8. Baker TL. Doing social research. 2nd ed. New York:McGraw-Hill; 1994.

9. Bowling A. Research methods in health:investigating health and health services.Buckingham: Open University Press; 1997.

10. Flick U. An introduction to qualitative research.London: Sage; 1998.

11. Denzin NK. The research act: a theoreticalintroduction to sociological methods. New York:McGraw-Hill; 1978.

12. Sapsford R, Abbott P. Research methods for nursesand the caring professions. Buckingham: OpenUniversity Press; 1992.

13. Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ2000;320:50–2.

14. Edwards SJL, Lilford RJ, Kiauka S. Different types ofsystematic review in health services research. In:Black N, Brazier J, Fitzpatrick R, Reeves B, editors.Health services research methods: a guide to bestpractice. London: BMJ Books; 1998. p. 255–9.

15. Hutton JL, Ashcroft R. What does “systematic”mean for reviews of methods? In: Black N, Brazier J,Fitzpatrick R, Reeves B, editors. Health servicesresearch methods: a guide to best practice. London:BMJ Books; 1998. p. 249–54.

16. Bowling A, Jacobson B, Southgate L. Explorationsin consultation of the public and health pro-fessionals on priority setting in an inner Londonhealth district. Soc Sci Med 1993;37:851–7.

17. Groves T. Public disagrees with professionals overNHS rationing [News]. BMJ 1993;306:673.

18. Furnham A, Meader N, McClelland A. Factorsaffecting nonmedical participants’ allocation of scarce medical resources. J Soc Behav Pers1998;13:735–46.

19. Widmark-Petersson V, von Essen L, Sjoden PO.Cancer patient and staff perceptions of caring andclinical care in free versus forced choice responseformats. Scand J Caring Sci 1998;12:238–45.

20. Jacobson B, Bowling A. Involving the public:practical and ethical issues [review]. Br Med Bull1995;51:869–75.

21. Rosko MD, McKenna W. Modeling consumerchoices of health plans: a comparison of twotechniques. Soc Sci Med 1983;17:421–9.

22. Kinnunen J, Lammintakanen J, Myllykangas M,Ryynanen OP, Takala J. Health care priorities as aproblem of local resource allocation. Int J HealthPlann Manage 1998;13:216–29.

23. Ryynanen OP, Myllykangas M, Niemela P, Kinnunen J, Takala J. Attitudes to prioritization in selected health care activities. Scand J Soc Welfare1998;7:320–9.

24. Angermeyer MC, Matschinger H, RiedelHeller SG.Whom to ask for help in case of a mental disorder?Preferences of the lay public. Soc Psychiatry PsychiatrEpidemiol 1999;34:202–10.

25. Furnham A. Factors relating to the allocation ofmedical resources. J Soc Behav Pers 1996;11:615–24.

26. Furnham A, Briggs J. Ethical ideology and moralchoice: a study concerning the allocation ofmedical resources. J Soc Behav Pers 1993;8:87–98.

27. Tolley K, Whynes D. Rationing in the NHS: settingpriorities depends on who you ask and how you askit. Paper presented to the UK Health Economists’Study Group meeting; 1994 Jul; Newcastle.

28. Bryson N, Ngwenyama OK, Mobolurin A. Aqualitative discriminant process for scoring andranking in group support systems. InformationProcess Manage 1994;30:389–405.

29. Bryson N. Supporting consensus formation ingroup support systems using the qualitativediscriminant process. Ann Oper Res 1997;71:75–91.

References

References

76

30. Ngwenyama OK, Bryson N. Generating belieffunctions from qualitative preferences: an approach to eliciting expert judgments andderiving probability functions. Data Knowledge Eng 1998;28:145–59.

31. Lancaster K. Consumer demand: a new approach.New York: Columbia University Press; 1971.

32. Luce D, Tukey J. Simultaneous conjointmeasurement: a new type of fundamentalmeasurement. J Math Psycho 1964;1:1–27.

33. Cattin P, Wittink D. Commercial use of conjointanalysis: a survey. J Marketing 1982;46:44–53.

34. Journal of Transport Economics and Policy 1988;22(1).

35. Rae D. Visibility impairment at Mesa VerdeNational Park: an analysis of benefits and costs ofcontrolling emissions in the Four Corners area.Report prepared for Electric Power ResearchInstitute. Boston, MA: Charles River Associates;1981.

36. Rae D. Benefits of improving visibility at GreatSmoky National Park. Draft report prepared forElectric Power Research Institute. Boston, MA.Charles River Associates; 1981.

37. Chinburapa V, Larson LN, Brucks M, Draugalis J,Bootman JL, Puto CP. Physician prescribingdecisions: the effects of situational involvement and task complexity on information acquisition and decision making. Soc Sci Med 1993;36:1473–82.

38. Gelb GM, Gelb BD. Physicians and hospitaldecision making: a two-stage technique forimprovement. Hosp Health Serv Adm 1987;32:139–49.

39. McClain JO, Rao VR. Trade-offs and conflicts inevaluation of health system alternatives: method-ology for analysis. Health Serv Res 1974;9:35–52.

40. Nickerson CA, McClelland GH, Petersen DM.Measuring contraceptive values: an alternativeapproach. J Behav Med 1991;14:241–66.

41. Orkin FK, Greenhow DE. A study of decisionmaking. Anesthesiology 1978;48:267–71.

42. Parker BR, Srinivasan V. A consumer preferenceapproach to the planning of rural primary health-care facilities. Oper Res 1976;24:991–1025.

43. Rosko MD, Walker LR, McKenna W, DeVita M.Measuring consumer preferences for ambulatorymedical care arrangements. J Med Syst1983;7:545–54.

44. Shemwell DJ, Yavas U. Congregate care facilityselection: a conjoint approach. Health Marketing Q1997;14(4):109–20.

45. Carroll NV, Gagnon JP. Consumer demand forpatient-oriented pharmacy services. Am J PublicHealth 1984;74:609–11.

46. Singh J, Cuttler L, Shin M, Silvers JB, Neuhauser D.Medical decision-making and the patient: under-standing preference patterns for growth hormonetherapy using conjoint analysis. Med Care 1998;36Suppl 8:AS31–45.

47. Stevens SS. A metric for social consensus. Science1966;151:530–41.

48. Armstrong RA, Brickley MR, Shepherd JP, Kay EJ.Healthy decision making: a new approach in healthpromotion using health state utilities. CommunityDent Health 1995;12:8–11.

49. Brazier J, Deverill M. The use of health relatedquality of life measures in economic evaluation. In:Black N, Brazier J, Fitzpatrick R, Reeves B, editors.Health services research methods: a guide to bestpractice. London: BMJ Books; 1998. p. 23–34.

50. Williams A. Economics of coronary bypass grafting.BMJ 1985;291:326–9.

51. Bunch WH, Chapman RG. Patient preferences in surgery for scoliosis. J Bone Joint Surg Am1985;67:794–9.

52. Chakraborty G, Gaeth GJ, Cunningham M. Under-standing consumers’ preferences for dental service[review]. J Health Care Mark 1993;13(3):48–58.

53. Chinburapa V, Larson LN. Predicting prescribingintention and assessing drug attribute importanceusing conjoint analysis. J Pharm Mark Manage1988;3(2):3–18.

54. Diamond JJ, Ruth DH, Markham FW, Rabinowitz HK, Rosenthal MP. Speciality selections of Jefferson Medical College students.Educ Health Prof 1994;17:322–8.

55. Geiger CJ, Wyse BW, Parent CR, Hansen RG.Nutrition labels in bar graph format deemed mostuseful for consumer purchase decisions usingadaptive conjoint analysis. J Am Diet Assoc1991;91:800–7.

56. Graf MA, Tanner DD, Swinyard WR. Optimizing the delivery of patient and physician satisfaction: aconjoint analysis approach. Health Care Manage Rev1993;18(4):34–43.

57. Harrison DD, Cooke CW. An elucidation of factors influencing physicians’ willingness toperform elective female sterilization. Obstet Gynecol 1988;72:565–70.

58. Harwood RH, Rogers A, Dickinson E, Ebrahim S.Measuring handicap: the London Handicap Scale, a new outcome measure for chronic disease. QualHealth Care 1994;3:11–16.

59. Reardon G, Pathak DS. Segmenting theantihistamine market: an investigation of consumerpreferences. J Health Care Mark 1990;10(3):23–33.


77

60. Ryan M, Shackley P, McIntosh E. Using conjointanalysis to elicit the views of health service users: anapplication to the patient health card. HealthExpectations 1998;1:117–29.

61. Ryan M, McIntosh E, Shackley P. Methodologicalissues in the application of conjoint analysis inhealth care. Health Econ Lett 1998;7:373–8.

62. Szeinbach SL, Mason HL, Schondelmeyer SN,Collins PD. Variables affecting pharmacists’willingness to accept third-party prescriptioncontracts: a conjoint analysis. J Health Care Mark1990;10(3):45–50.

63. van der Pol M, Ryan M. Using conjoint analysis to establish consumer preferences for fruit andvegetables. Br Food J 1996;98(8):5–12.

64. Wigton RS, Hoellerich VL, Patil KD. Howphysicians use clinical information in diagnosingpulmonary embolism: an application of conjointanalysis. Med Decis Making 1986;6:2–11.

65. Wind Y, Spitz LK. Analytical approach to marketingdecisions in health-care organisations. Oper Res1976;24:973–90.

66. Richardson DK, Gabbe SG, Wind Y. Decisionanalysis of high-risk patient referral. Obstet Gynecol1984;63:496–501.

67. Coen R, O’Mahoney D, O’Boyle C, Joyce CRB,Hiltbrunner B, Walsh JB, et al. Measuring thequality of life of dementia patients using theschedule for the evaluation of individual quality of life. Ir J Psychol 1993;14:154–63.

68. O’Boyle CA, McGee H, Hickey A, O’Malley K, Joyce CRB. Individual quality of life in patientsundergoing hip replacement. Lancet1992;339:1088–91.

69. Browne JP, O’Boyle A, McGee HM, Joyce RB,McDonald NJ, O’Malley K, et al. Individual qualityof life in the healthy elderly. Qual Life Res1994;3:235–44.

70. McGee HM, O’Boyle CA, Hickey A, O’Malley K,Joyce CRB. Assessing the quality of life of theindividual: the SEIQoL with a healthy and agastroenterology unit population. Psychol Med1998;21:749–59.

71. Oppenheim AN. Questionnaire design,interviewing and attitude measurement. London: Pinter; 1992.

72. Aldrich JH, Niemi RG, Rabinowitz G, Rohde DW.The measurement of public-opinion about public-policy – a report on some new issue questionformats. Am J Political Sci 1982;26:391–414.

73. Richardson A, Charny M, Hanmerlloyd S. Public-opinion and purchasing. BMJ 1992;304:680–2.

74. Lahti S, Tuutti H, Hausen H, Kaariainen R.Opinions of different subgroups of dentists andpatients about the ideal dentist and the idealpatient. Community Dent Oral Epidemiol1995;23:89–94.

75. Weekman VW. Laboratory reactors and theirlimitations. Am Instit Chem Eng J 1974;20:833–40.

76. Borowsky SJ, Davis MK, Goertz C, Lurie N. Are allhealth plans created equal? The physician’s view.JAMA 1997;278:917–21.

77. Tymstra T, Andela M. Opinion of Dutch physicians,nurses and citizens on health care policy, rationingand technology. JAMA 1993;270:2995–9.

78. Lahti S, Tuutti H, Hausen H, Kaariainen R.Comparison of ideal and actual behavior of patients and dentists during dental treatment.Community Dent Oral Epidemiol 1995;23:374–8.

79. Lahti S, Tuutti H, Hausen H, Kaariainen R. Dentistand patient opinions about the ideal dentist andpatient – developing a compact questionnaire.Community Dent Oral Epidemiol 1992;20:229–34.

80. Marks GB, Dunn SM, Woolcock AJ. A scale for the measurement of quality-of-life in adults withasthma. J Clin Epidemiol 1992;45:461–72.

81. Conway T, Hu TC, Harrington T. Setting healthpriorities: community boards accurately reflect the preferences of the community’s residents. J Community Health 1997;22:57–68.

82. Checkoway B. Public participation in healthplanning agencies: promise and practice. J HealthPolit Policy Law 1982;7:723–33.

83. Pfennings L, Cohen L, Vanderploeg H.Preconditions for sensitivity in measuring change –visual analog scales compared to rating-scales in alikert format. Psychol Rep 1995;77:475–80.

84. Wilson JR, Fazey JA. Self-esteem, compliance, andcervical screening. Psychol Rep 1995;77:891–8.

85. Holmes C. A statistical evaluation of rating scales. J Mark Res Soc 1974;16:87–107.

86. McKennell AC, Bryner JN. Self images and smokingbehaviour among school boys. Br J Educ Psychol1969;39:27–39.

87. Valois P, Godin G. The importance of selectingappropriate adjective pairs for measuring attitudebased on the semantic differential method. QualQuantity 1991;25:57–68.

88. Wilbur J, Miller A, Montgomery A. The influence of demographic characteristics, menopausal status,and symptoms on women’s attitudes towardmenopause. Women Health 1995;23:19–39.

89. Bowles C. Measure of attitude toward menopauseusing the semantic differential model. Nursing Res1986;35:81–5.

References

78

90. Wikblad KF, Wibell LB, Montin KR. The patient’sexperience of diabetes and its treatment –construction of an attitude scale by a semanticdifferential technique. J Adv Nurs 1990;15:1083–91.

91. Nichols BS, Misra R, Alexy B. Cancer detection:How effective is public education? Cancer Nurs1996;19:98–103.

92. Swain R, McNamara M. The effects of aparticipative programme on Irish pupils’ attitudesto HIV/AIDS. Health Educ Res 1997;12:267–73.

93. Girón M, Gomezbeneyto M. Relationship betweenfamily attitudes measured by the semantic differ-ential and relapse in schizophrenia – a 2-yearfollow-up prospective-study. Psychol Med1995;25:365–71.

94. Wierenga B, Ophuis PAMO, Huizingh EKR,Vancampen PAFM. Hierarchical scaling ofmarketing decision-support systems. Decis Support Syst 1994;12:219–32.

95. Kelloway EK, Barling J. Members’ participation inlocal union activities – measurement, prediction,and replication. J Appl Psychol 1993;78:262–79.

96. Katz YJ, Schmida M. A guttman scale factorstructure of comprehensiveness. Educ Psychol Meas1993;53:225–32.

97. Podell L, Perkins JC. A Guttman scale for sexualexperience – a methodological note. J AbnormPsychol 1957;54:420–2.

98. Edmundson EW, Koch WR, Silverman S. A facetanalysis approach to content and construct-validity.Educ Psychol Meas 1993;53:351–68.

99. Santos ML, Booth DA. Influences on meatavoidance among British students. Appetite1996;27:197–205.

100. Sanner M. A comparison of public-attitudes towardautopsy, organ donation, and anatomic dissection –a Swedish survey. JAMA 1994;271:284–8.

101. Oshea E, Murray P. Care provision and dependencyin long-stay institutions. Econ Soc Rev 1997;28:43–61.

102. Petersen PE. Guttman scale analysis of dental-healthattitudes and knowledge. Community Dent OralEpidemiol 1989;17:170–2.

103. Khayat K, Salter B. Patient satisfaction surveys as amarket-research tool for general practices. Br J GenPract 1994;44:215–19.

104. Ware JE, Hays RD. Methods for measuring patientsatisfaction with specific medical encounters. MedCare 1988;26:393–402.

105. Makhdoomn YM, Elzubier AG, Hanif M.Satisfaction with health care among primary health care centers attendees’ in Al-Khobar, Saudi Arabia. Saudi Med J 1997;18:227–30.

106. Chiu L. Family caregiver’s satisfaction with homecare in the Taipei Metropolitan Area. Public HealthNurs 1997;14:42–50.

107. Sharma RD, Chahal H. Patient satisfaction in publichealth system – a case study. Indian J Soc Work1995;56:445–56.

108. Cohen G, Forbes J, Garraway M. Can differentpatient satisfaction survey methods yield consistentresults? Comparison of three surveys. BMJ1996;313:841–4.

109. Grogan S, Conner M, Willits D, Norman P.Development of a questionnaire to measurepatients’ satisfaction with general practitioners’services. Br J Gen Pract 1995;45:525–9.

110. Loeken K, Steine S, Sandvik L, Laerum E, Finset A.A new measure of patient satisfaction with mammo-graphy. Validation by factor analytic technique. Fam Pract 1996;13:67–74.

111. Avis M, Bond M, Arthur A. Questioning patientsatisfaction: an empirical investigation in twooutpatient clinics. Soc Sci Med 1997;44:85–92.

112. Counte MA. An examination of the convergentvalidity of three measures of patient satisfaction inan outpatient center. J Chronic Dis 1979;32:583–8.

113. Finkelstein BS, Harper DL, Rosenthal GE. Patientassessments of hospital maternity care: a useful toolfor consumers? Health Serv Res 1999;34:623–40.

114. Ruggeri M, Dall’Agnola R. The development anduse of the Verona Expectations for Care Scale(VECS) and the Verona Service Satisfaction Scale(VSSS) for measuring expectations and satisfactionwith community-based psychiatric services inpatients, relatives and professionals. Psychol Med1993;23:511–23.

115. Han SH, Jung ES, Jung MY, Kwak JY, Park SJ, Choe JH. A psychophysical evaluation of interior-design alternatives for a high-speed train. ComputInd Eng 1994;27:397–400.

116. Westra BL, Cullen L, Brody D, Jump P, Geanon L,Milad E. Development of the home care clientsatisfaction instrument. Public Health Nurs1995;12:393–9.

117. Wilde B, Larsson G, Larsson M, Starrin B. Quality of care – development of a patient-centered questionnaire based on a groundedtheory model. Scand J Caring Sci 1994;8:39–48.

118. Holmes-Rovner M, Kroll J, Schmitt N, Rovner DR,Breer ML, Rothert ML, et al. Patient satisfactionwith health care decisions: the satisfaction withdecision scale. Med Decis Making 1996;16:58–64.


79

119. Guyatt GH, Mitchell A, Molloy DW, Capretta R,Horsman J, Griffith L. Measuring patient andrelative satisfaction with level or aggressiveness of care and involvement in care decisions in thecontext of life threatening illness. J Clin Epidemiol1995;48:1215–24.

120. Fitzpatrick R. Surveys of patient satisfaction:II–Designing a questionnaire and conducting asurvey. BMJ 1991;302:1129–32.

121. Campbell D, Christopher K. The evolution of apatient satisfaction survey. Dimens Health Serv1991;68:18–20.

122. Van Campen C, Sixma H, Friele RD, Kerssens JJ,Peters L. Quality of care and patient satisfaction: areview of measuring instruments [review]. Med CareRes Rev 1995;52:109–33.

123. Bamford C, Jacoby A. Development of patientsatisfaction questionnaires: 1. Methodologicalissues. Qual Health Care 1992;1:153–7.

124. Roberts RE, Pascoe GC, Attkisson CC. Relationship of service satisfaction to lifesatisfaction and perceived well-being. Eval Program Plann 1983;6:373–83.

125. Pascoe GC, Attkisson CC, Roberts RE. Comparison of indirect and direct approaches to measuring patient satisfaction. Eval ProgramPlann 1983;6:359–71.

126. Parker SC, Kroboth FJ. Practical problems ofconducting patient-satisfaction surveys. J Gen Intern Med 1991;6:430–5.

127. McCusker J. Development of scales to measuresatisfaction and preferences regarding long-termand terminal care. Med Care 1984;22:476–93.

128. Damkot DK, Pandiani JA, Gordon LR.Development, implementation, and findings of a continuing client satisfaction survey. Community Mental Health J 1983;19:265–78.

129. Hinshaw AS, Atwood JR. A Patient SatisfactionInstrument: precision by replication. Nurs Res1982;31:170–5.

130. Ware JEJ. How to survey patient satisfaction. DrugIntell Clin Pharm 1981;15:892–9.

131. Hines BL, Clarkson QD, Smith DD. Developmentand use of a patient satisfaction questionnaire. J Fam Pract 1977;4:148–9.

132. Zyzanski SJ, Hulka BS, Cassel JC. Scale for themeasurement of “satisfaction” with medical care:modifications in content, format and scoring. Med Care 1974;12:611–20.

133. Harrington V, Lackey NR, Gates MF. Needs ofcaregivers of clinic and hospice cancer patients.Cancer Nurs 1996;19:118–25.

134. Bredart A, Razavi D, Delvaux N, Goodman V,Farvacques C, VanHeer C. A comprehensiveassessment of satisfaction with care for cancerpatients. Support Care Cancer 1998;6:518–23.

135. Martin PD, Brantley PJ, McKnight GT, Jones GN,Springer A. The multidisciplinary hemodialysispatient satisfaction scale: reliability, validity, andscale development. Assessment 1997;4:95–105.

136. Singh J, Wood VR, Goolsby J. Consumers’satisfaction with health care delivery: issues ofmeasurement, issues of research design [review]. J Ambul Care Mark 1990;4(1):105–15.

137. Cockburn J, Hill D, Irwig L, De Luise T, Turnbull D,Schofield P. Development and validation of aninstrument to measure satisfaction of participants at breast screening programmes. Eur J Cancer1991;27:827–31.

138. Ford RC, Bach SA, Fottler MD. Methods ofmeasuring patient satisfaction in health careorganizations [review]. Health Care Manage Rev1997;22:74–89.

139. Batchelor C, Owens DJ, Read M. Patient satisfaction studies: methodology, management and consumer evaluation. Int J Health Care QualAssur 1994;7(7):22–30.

140. Niedz BA. Correlates of hospitalized patients’perceptions of service quality. Res Nurs Health1998;21:339–49.

141. Hugo B, Becker S, Witt E. Assessment of thecombined orthodontic-surgical treatment from the patients’ point of view. A longitudinal study. J Orofacial Orthop 1996;57:88–101.

142. Dansky KH, Brannon D. Discriminant analysis: atechnique for adding value to patient satisfactionsurveys. Hosp Health Serv Adm 1996;41:503–13.

143. Scott A, Smith RD. Keeping the customer satisfied:issues in the interpretation and use of patientsatisfaction surveys. Int J Qual Health Care1994;6:353–9.

144. Fitzpatrick R. Surveys of patient satisfaction: 1 – important general considerations. BMJ1991;302:887–9.

145. Carr-Hill RA. The measurement of patientsatisfaction. J Public Health Med 1992;14:236–49.

146. Baker R. The reliability and criterion validity of ameasure of patients’ satisfaction with their generalpractice. Fam Pract 1991;8:171–7.

147. Baker R. Development of a questionnaire to assesspatients’ satisfaction with consultations in generalpractice. Br J Gen Pract 1990;40:487–490.

References

80

148. McKinley RK, Manku-Scott T, Hastings AM, French DP, Baker R. Reliability and validity of a new measure of patient satisfaction with out of hours primary medical care in the UnitedKingdom: development of a patient questionnaire.BMJ 1997;314:193–8.

149. Kalman TP. An overview of patient satisfaction with psychiatric treatment. Hosp CommunityPsychiatry 1983;34:48–54.

150. Tarnowski KJ, Simonian SJ. Assessing treatmentacceptance: the abbreviated acceptability ratingprofile. J Behav Ther Exp Psychiatry 1992;23:101–6.

151. Essex DW, Fox JA, Groom JM. The development,factor analysis, and revision of a client satisfactionform. Community Mental Health J 1981;17:226–35.

152. Hulka BS, Zyzanski SJ. Validation of a patientsatisfaction scale: theory, methods and practice. Med Care 1982;20:649–53.

153. Nguyen TD, Attkisson CC, Stegner BL. Assessmentof patient satisfaction: development of a serviceevaluation questionnaire. Eval Program Plann1983;6:299–313.

154. Gonzalves PE, Minderler JJ, Tompkins DL. Apatient satisfaction survey: a basis for changingdelivery of services. Mil Med 1995;160:486–8.

155. Dull VT, Lansky D, Davis N. Evaluating a patientsatisfaction survey for maximum benefit. Jt Comm J Qual Improv 1994;20:444–53.

156. Baker R, Whitfield M. Measuring patientsatisfaction: a test of construct validity. Qual Health Care 1992;1:104–9.

157. Pope C, Mays N. Qualitative research: reaching theparts other methods cannot reach. An introductionto qualitative methods in health and health servicesresearch. BMJ 1995;311:42–5.

158. Sabourin S, Wright J, Duchesne C, Belisle S. Areconsumers of modern fertility treatments satisfied?Fertil Steril 1991;56:1084–90.

159. Bisset A, Chesson R. Is this satisfaction surveysatisfactory? Some points to consider in theirplanning and assessment. Health Bull 2000;58:45–52.

160. Green J. On the receiving end. Health Serv J1988;98:880–2.

161. Wolf M, Putman S, James S, Stiles W. The MedicalInterview Satisfaction Scale: development of a scaleto measure patient perceptions of physicianbehaviour. J Behav Med 1978;1:391–401.

162. Eardley A, Lancaster G, Elkind A. Made to measuresurvey. Health Serv J 1990;100:1773.

163. Garcia J, Redshaw M, Fitzsimons B, Keene J. FirstClass Delivery: a national survey of women’s views ofmaternity care. Oxford: Audit Commission; 1998.

164. Scottish Programme for Clinical Effectiveness inReproductive Health, Dugald Baird Centre forResearch On Womens’ Health. Maternity carematters: an audit of maternity services in Scotland1998. Aberdeen: SPCERH; 1999. SPCERHPublication No. 9.

165. Parasuraman A, Zeithaml VA, Berry LL. Servqual –a multiple-item scale for measuring consumerperceptions of service quality. J Retailing1988;64:12–40.

166. Camilleri D, O’Callaghan M. Comparing public and private hospital care service quality. Int J HealthCare Qual Assur 1998;11(4–5):127–33.

167. Scardina SA. SERVQUAL: a tool for evaluatingpatient satisfaction with nursing care. J Nurs CareQual 1994;8(2):38–46.

168. Sargeant A, Kaehler J. Factors of patient satisfactionwith medical services: the case of GP practices inthe UK. Health Mark Q 1998;16(1):55–77.

169. Kaldenberg D, Becker BW, Browne BA, Browne WG. Identifying service quality strengthsand weaknesses using SERVQUAL: a study of dental services. Health Mark Q 1997;15(2):69–86.

170. Sewell N. Continuous quality improvement in acute health care: creating a holistic and integratedapproach. Int J Health Care Qual Assur1997;10(1):20–6.

171. Youssef FN, Nel D, Bovaird T. Health care quality in NHS hospitals. Int J Health Care Qual Assur1996;9(1):15–28.

172. Anderson EA. Measuring service quality at auniversity health clinic. Int J Health Care Qual Assur1995;8(2):32–7.

173. Headley DE, Miller SJ. Measuring service qualityand its relationship to future consumer behavior. J Health Care Mark 1993;13(4):32–41.

174. Walbridge SW, Delene LM. Measuring physicianattitudes of service quality. J Health Care Mark1993;13(1):6–15.

175. Soliman AA. Assessing the quality of health care: a consumerist approach. Health Mark Q1992;10(1–2):121–41.

176. Reidenbach RE, Sandifer-Smallwood B. Exploringperceptions of hospital operations by a modifiedSERVQUAL approach. J Health Care Mark1990;10(4):47–55.

177. Lam SSK. SERVQUAL: a tool for measuringpatients’ opinions of hospital service quality inHong Kong. Total Qual Manage 1997;8:145–52.

178. Duffy JA, Duffy M, Kilbourne W. Cross nationalstudy of perceived service quality in long-term carefacilities. J Aging Stud 1997;11:327–36.


81

179. Chaston I. A comparative-study of internal customermanagement practices within service sector firmsand the National Health Service. J Adv Nurs1994;19:299–308.

180. Mitchell R, Leanna JC, Hyde R. Client satisfactionwith nursing services: evaluation in an occupationalhealth setting. AAOHN J 1999;47(2):74–8.

181. Raspollini E, Pappalettera M, Riccardi D,Parravicini A, Sestili S, Rebulla P, et al. Use ofSERVQUAL to assess clinicians’ satisfaction with the blood transfusion service. Vox Sang1997;73:162–6.

182. Duffy J-AM, Ketchand AA. Examining the role of service quality in overall service satisfaction. J Manage Issues 1998;10(2):240–55.

183. Charny MC, Lewis PA, Farrow SC. Choosing whoshall not be treated in the NHS. Soc Sci Med1989;28:1331–8.

184. Lewis PA, Charny M. Which of two individuals doyou treat when only their ages are different and youcan’t treat both? J Med Ethics 1999;15:28–32.

185. Mooney G, Jan S, Wiseman V. Examiningpreferences for allocating health care gains. Health Care Anal 1995;3:261–5.

186. Ryynanen O-P, Myllykangas M, Vaskilampi T, Takala J. Random paired scenarios – a method for investigating attitudes to prioritisation inmedicine. J Med Ethics 1996;22:238–42.

187. Bryan S, Buxton M, Sheldon R, Grant A. Magneticresonance imaging for the investigation of kneeinjuries: an investigation of preferences. Health Econ1998;7:595–603.

188. Chakraborty G, Ettenson R, Gaeth G. Howconsumers choose health insurance. J Health CareMark 1994;14(1):21–33.

189. Farrar S, Ryan M. Response-ordering effects: a methodological issue in conjoint analysis. Health Econ 1999;8:75–9.

190. Ryan M. A role for conjoint analysis in technologyassessment in health care. Int J Technol Assess HealthCare 1999;15:443–57.

191. Ferguson RP, Wetle T, Dubitzky D, Winsemius D.Relative importance to elderly patients of effective-ness, adverse effects, convenience and cost ofantihyperintensive medications. Drugs Aging1994;4:56–62.

192. Hakim Z, Pathak DS. Modelling the EuroQol data:a comparison of discrete choice conjoint andconditional preference modelling. Health Econ1999;8:103–16.

193. McIntosh E, Ryan M. The theoretical basis ofdiscrete choice conjoint analysis: testing fortransitivity and continuity within an empiricalapplication in health care. Paper presented to the UK Health Economists’ Study Group Meeting; 1999 Jul; Aberdeen.

194. Ryan M. Measuring benefits in health care: the role of discrete choice conjoint analysis. Paperpresented to the 2nd International Conference ofthe International Health Economics’ Association;1999 Jun 7–9; Rotterdam.

195. Propper C. Contingent valuation of time spent onNHS waiting lists. Econ J 1990;100:193–9.

196. Ratcliffe J, Buxton M. Patients’ preferencesregarding the process and outcomes of life. Int JTechnol Assess Health Care 1999;15:340–51.

197. Ryan M. Using conjoint analysis to take account of patient preferences and go beyond healthoutcomes: an application to in vitro fertilisation. Soc Sci Med 1999;48:535–46.

198. Ryan M. A role for conjoint analysis in technologyassessment in health care? Int J Technol Assess HealthCare 1999;15:443–57.

199. Ryan M, Hughes J. Using conjoint analysis to assesswomen’s preferences for miscarriage management.Health Econ 1997;6:261–73.

200. Ryan M, Wordsworth S. Sensitivity of willingness topay estimates to the level of attributes in discretechoice experiments. Scott J Polit Econ2000;47:504–24.

201. San Miguel F, Ryan M, McIntosh E. Demonstratingthe use of conjoint analysis in economic evalu-ations: an application to menorrhagia. Appl Econ2000;32:823–33.

202. Scott A, Vick S. An application of principle–agenttheory to the doctor–patient relationship. Scott JPolit Econ 1999;46:111–34.

203. Vick S, Scott A. Agency in health care. Examiningpatients’ preferences for attributes. J Health Econ1998;17:587–605.

204. Hadorn DC, Hays RD, Uebersax J, Hauber T.Improving task comprehension in the measurementof health state preferences. A trial of informationalcartoon figures and a paired-comparison task. J ClinEpidemiol 1992;45:233–43.

205. Ryan M, Farrar S. Eliciting preferences for healthcare using conjoint analysis. BMJ 2000;320:1530–3.

206. Spoth R, Redmond C. Identifying programpreferences through conjoint analysis: illustrativeresults from a parent sample. Am J Health Promot1993;8:124–33.

207. Van der Pol M, Cairns J. Establishing patientpreferences for blood transfusion support: anapplication of conjoint analysis. J Health Serv ResPolicy 1998;3:70–6.

References

82

208. Spoth R, Redmond C, Ball A. Stage of quitting andmotivational factors relevant to smoking cessationprogram choices. Psychol Addict Behav 1993;7:29–42.

209. Spoth R. Multi-attribute analysis of benefitmanagers’ preferences for smoking cessationprograms. Health Values 1990;14(5):3–15.

210. Hornberger JC, Habraken H, Bloch DA. Minimumdata needed on patient preferences for accurate,efficient medical decision making. Med Care1995;33:297–310.

211. Farrar S, Ryan M, Ross D, Ludbrook A. Usingdiscrete choice modelling in priority setting: anapplication to clinical service developments. Soc Sci Med 2000;50:63–75.

212. Verhoef CG, Maas A, Stalpers LJA, Verbeek ALM,Wobbes T, van Daal WAJ. The feasibility of additiveconjoint measurement in measuring utilities inbreast cancer patients. Health Policy 1991;17:39–50.

213. Maas A, Stalpers L. Assessing utilities by means ofconjoint measurement: an application in medicaldecision analysis. Med Decis Making 1992;12:288–97.

214. Rangone A. An analytical hierarchy processframework for comparing the overall performanceof manufacturing departments. Int J Oper ProdManage 1996;16(8):104–19.

215. Saaty RW. The analytic hierarchy process – what it isand how it is used. Math Model 1987;9:161–76.

216. Mulye R. An empirical comparison of three variantsof the AHP and two variants of conjoint analysis. J Behav Decis Making 1998;11:263–80.

217. Golden BL, Wasil EA, Harker PT. The analytichierarchy process: applications and studies. New York: Springer-Verlag; 1989.

218. Harker PT, Vargas LG. The theory of ratio scaleestimation – saaty’s analytic hierarchy process.Management Sci 1987;33:1383–403.

219. Saaty TL. Axiomatic foundation of the analytichierarchy process. Management Sci 1986;32:841–55.

220. Hanratty PJ, Joseph B. Decision-making inchemical-engineering and expert systems – appli-cation of the analytic hierarchy process to reactorselection. Comput Chem Eng 1992;16:849–60.

221. Mohanty RP, Venkataraman S. Use of analytichierarchy process for selecting automated manu-facturing systems. Int J Oper Prod Manage1993;13(8):45–57.

222. Olson DL, Venkataramanan M, Mote JL. Atechnique using analytical hierarchy process inmultiobjective planning-models. Socioecon Plann Sci 1986;20:361–8.

223. Saaty TL. A scaling method for priorities inhierarchical structures. J Math Psychol1977;15:234–81.

224. Mohanty RP, Deshmukh SG. Use of analytichierarchy process for evaluating sources of supply. Int J Phys Distribution Logistics Manage1993;23(3):22–8.

225. Schwartz RG, Oren S. Using analytic hierarchies for consumer research and market modelling. Math Comput Model 1988;11:266–71.

226. Dolan JG. Choosing initial antibiotic therapy foracute pyleonephritis. In: Golden BLWEA, HarkerPT, editors. The analytic hierarchy process:applications and studies. London: Springer-Verlag;1989. p. 213–24.

227. Dolan JG, Isselhardt BJ, Cappuccio JD. The analytichierarchy process in medical decision making: atutorial. Med Decis Making 1989;9:40–50.

228. Dolan JG. Medical decision making using theanalytic hierarchy process: choice of initialantimicrobial therapy for acute pyelonephritis. Med Decis Making 1999;9:51–6.

229. Dolan JG, Bordley DR. Isoniazid prophylaxis: theimportance of individual values. Med Decis Making1994;14:1–8.

230. Dolan JG, Bordley DR. Should concern over gastriccancer influence the choice of diagnostic tests inpatients with acute upper gastrointestinal bleeding.Proceedings of the 2nd International Symposiumon the Analytic Hierarchy Process; 1991 Aug 12–14;Pittsburgh (PA). p. 391–403.

231. Dolan JG. Promoting decision making partnershipsusing the Analytic Hierarchy Process. In: SuchmanAL, Botelho RJ, Hinton-Walker P, editors.Partnerships in health care: transforming relationalprocess. Rochester, NY: University of RochesterPress; 1998. p. 151–66.

232. Dolan JG. Diagnostic strategies in the managementof acute upper gastrointestinal bleeding: patientand physician preferences. J Gen Intern Med1993;8:525–9.

233. Dolan JG, Bordley DR. Using the Analytic hierarchyProcess (AHP) to develop and disseminateguidelines. Q Rev Bull 1992;18:440–7.

234. Dolan JG. Clinical decision making using theanalytic hierachy process: choice of antibiotictreatment for community-acquired pyelonephritis[abstract]. Clin Res 1987;35:738.

235. Dolan JG. Are patients capable of using the analytichierarchy process and willing to use it to help makeclinical decisions. Med Decis Making 1995;15:76–80.

236. Dolan JG. Involving patients in complex decisionsabout their care: an approach using the analytichierarchy process. J Gen Intern Med 1993;8:204–9.

237. Dolan JG. Can decision analysis adequatelyrepresent clinical problems? J Clin Epidemiol1990;43:277–84.


83

238. Lusk EJ. Analysis of hospital capital decisionalternatives: a priority assessment model. J Oper Res Soc 1979;30:439–48.

239. Javalgi RG, Rao SR, Thomas EG. Choosing ahospital: analysis of consumer tradeoffs. J HealthCare Mark 1989;11(1):12–22.

240. Saaty TL. The analytic hierarchy process and healthcare problems. In: Tilquin C, editor. Systemsscience in health care. Vol. 1. Proceedings of theInternational Conference on Systems Science inHealth Care; 1980 Jul; Montreal. Toronto:Pergamon Press; 1981. p. 147–58.

241. Hannan EL, O’Donnell J, Freedland T. A priorityassessment model for standards and conditions in a long term care survey. Socioecon Plann Sci1981;15:277–89.

242. Dougherty JJ, Saaty TL. Optimum determination of hospital requirements. In: Saaty TL, Vargas LG,editors. The logic of priorities: applications inbusiness, energy, health and transportation. Boston:Kluwer-Nijhoff Publishing; 1982. p. 165–81.

243. Odynocki B. The NHI proposals: an evaluation. In:Tilquin C, editor. Systems science in health care.Vol. 1. Proceedings of the International Conferenceon Systems Science in Health Care; 1980 Jul;Montreal. Toronto: Pergamon Press; 1981. p. 159–68.

244. Odynocki B. The forward and backward processesin health policy planning. Math Comput Simulation1983;25:146–55.

245. Gafni A. The standard gamble method: what isbeing measured and how it is interpreted. HealthServices Res 1994;29:207–24.

246. Bleichrodt H, Johannesson M. Standard gamble,time trade-off and rating scale: experimental resultson the ranking properties of QALYs. J Health Econ1997;16:155–75.

247. Torrance G, Thomas W, Sackett D. A utilitymaximisation model for evaluation of health careprograms. Health Serv Res 1972;7:118–33.

248. Patrick DL, Bush JW, Chen MM. Methods formeasuring levels of well-being for a health statusindex. Health Serv Res 1973;8:228–45.

249. Nord E. Methods for quality adjustment of life years[review]. Soc Sci Med 1992;34:559–69.

250. Green C. Assessing the societal value of health care:is the PTO up to the job? Paper presented to theUK Health Economists’ Study Group Meeting; 1999Jul; Aberdeen.

251. Nord E. The person-trade-off approach to valuinghealth care programs. Med Decis Making1995;15:201–8.

252. Prades JLP. Is the person trade-off a valid methodfor allocating health care resources? Health Econ1997;6:71–81.

253. Brazier J, Deverill M, Green C, Harper R, Booth A.A review of the use of health status measures ineconomic evaluation. Health Technol Assess1999;3(9).

254. Ubel PA, Loewenstein G, Scanlon D, Kamlet M.Value measurement in cost-utility analysis:explaining the discrepancy between rating scaleand person trade-off elicitations. Health Policy1998;43:33–44.

255. Nord E. The person trade-off approach to valuinghealth care programs. Fairfield, Vic: NationalCentre for Health Program Evaluation, MonashUniversity/University of Melbourne; 1994. CHPEWorking Paper 38.

256. Diener A, Obrien B, Gafni A. Health carecontingent valuation studies: a review andclassification of the literature. Health Econ1998;7:313–26.

257. Klose T. The contingent valuation method in healthcare. Health Policy 1999;47:97–123.

258. Olsen JA, Smith R. Who have been asked to valuewhat? A review of 54 WTP surveys on health andhealth care. Paper presented to the UK HealthEconomists’ Study Group Meeting; 1998 Jul;Galway.

259. Donaldson C, Jones AM, Mapp TJ, Olson JA.Limited dependent variables in willingness to paystudies: applications in health care. Appl Econ1998;30:667–77.

260. Ryan M, Ratcliffe J, Tucker J. Using willingness topay to value alternative models of antenatal care.Soc Sci Med 1997;44:371–80.

261. Johannesson M, Jonsson B, Borgquist L. Willingnessto pay for antihyperintensive therapy – results of aSwedish pilot study. J Health Econ 1991;10:461–74.

262. Olsen JA, Donaldson C. Helicopters, hearts andhips: using willingness to pay to set priorities forpublic sector health care programmes. Soc Sci Med1998;46:1–12.

263. Johannesson M, Johansson P-O, Kristrom B,Gerdtham U-G. Willingness to pay for antihyper-intensive therapy – further results. J Health Econ1993;12:95–108.

264. Churchman CW, Ackoff RL. An approximatemeasure of value. Oper Res 1954;2:172–81.

265. Dickinson AL. Getting the best from additionalresources. Hosp Health Serv Rev 1979;75:127–9.

266. Secker J. Qualitative methods in health promotionresearch: some criteria for quality. Health Educ J1995;54:74–87.

267. Hoinville G. The Priority Evaluator Method.London: Department of Social and CommunityPlanning and Research, University of London;1977. Methodological Working Paper 3.

References

84

268. Hoinville G. Evaluating community preferences. J Mark Res Soc 1996;38:483–504.

269. Ruta DA, Garratt AM, Leng M, Russell IT,MacDonald LM. A new approach to themeasurement of quality of life – the patientgenerated index. Med Care 1999;32:1109–26.

270. Macduff C, Russell E. The problem of measuringchange in individual health-related quality of life bypostal questionnaire: use of the patient-generatedindex in a disabled population. Qual Life Res1998;7:761–9.

271. Ruta DA, Garratt AM, Russell IT. Patient centredassessment of quality of life for patients with fourcommon conditions. Qual Health Care 1999;8:22–9.

272. Herd RM, Tidman MJ, Ruta DA, Hunter JA.Measurement of quality of life in atopic dermatitis:correlation and validation of two different methods.Br J Dermatol 1997;136:502–7.

273. Jenkinson C, Stradling J, Petersen S. How should we evaluate health status? A comparison of threemethods in patients presenting with obstructivesleep apnoea. Qual Life Res 1998;7:95–100.

274. Hickey AM, Bury G, O’Boyle CA, Bradley F, O’Kelly FD, Shannon W. A new short formindividual quality of life measure (SEIQoL-DW):application in a cohort of individuals withHIV/AIDS. BMJ 1996;313:29–33.

275. Browne JP, O’Boyle C, McGee HM, McDonald NJ,Joyce CRB. Development of a direct weightingprocedure for quality of life domains. Qual Life Res1997;6:301–9.

276. Clark TN. Community social indicators: fromanalytical models to policy applications. UrbanAffairs Q 1973;9:3–36.

277. Clark TM. Can you cut a budget pie? Policy Politics1974;3(2):3–31.

278. Hauser JR, Shugan SM. Intensity measures ofconsumer preference. Oper Res 1980;28:278–320.

279. Ham C, Locock L. International approaches topriority setting in health care: an annotated listingof official and semi-official publications, with aselection of key academic references. Birmingham:Health Services Management Centre, University ofBirmingham; 1998. HSMC Handbook Series no. 25.

280. Honigsbaum F, Richards J, Lockett T. Prioritysetting in action: purchasing dilemmas. Oxford:Radcliffe Medical Press; 1995.

281. Ratcliffe J. Distribution principles for the allocationof donor liver grafts: results from a ‘social’ conjointanalysis survey. Paper presented at the UK HealthEconomists’ Study Group Meeting; 1999 Jan;Birmingham.

282. Strauss RP, Hughes GD. A new approach to thedemand for public goods. J Public Econ1976;6:191–204.

283. Garland MJ. Rationing in public: Oregon’s priority-setting methodology. In: Strosberg MA, editor.Rationing America’s medical care: the Oregon planand beyond. Washington, DC: The BrookingsInstitution; 1992. p. 37–59.

284. Dixon J, Welch HG. Priority setting: lessons fromOregon. Lancet 1991;337:891–4.

285. Eddy DM. What’s going on in Oregon? JAMA1991;266:417–20.

286. Klein R. On the Oregon trail: rationing health care[editorial] BMJ 1991;302:1–2.

287. Foster G. Fishing with the net for research data. Br J Educ Technol 1994;25:91–7.

288. Ayanian JZ, Cleary PD, Weissman JS, Epstein AM.The effect of patients’ preferences on racialdifferences in access to renal transplantation. N Engl J Med 1999;341:1661–9.

289. Dicker A, Armstrong D. Patients’ views of prioritysetting in health care: an interview survey in onepractice. BMJ 1995;311:1137–9.

290. Williams B, Coyle J, Healy D. The meaning ofpatient satisfaction: an explanation of highreported levels. Soc Sci Med 1998;47:1351–9.

291. Crabtree B, Miller WL. A qualitative approach toprimary care research: the long interview. Fam Med1991;23:145–51.

292. Wilson K, Roe B, Wright L. Telephone or face-to-face interviews?: a decision made on the basis of apilot study. Int J Nurs Stud 1998;35:314–21.

293. Sohier R. The dyadic interview as a tool for nursingresearch. Appl Nurs Res 1995;8:96–101.

294. Guest JF, Hart WM, Cookson RF. Cost analysis ofpalliative care for terminally ill cancer patients inthe UK after switching from weak to strong opoids.Pharmacoeconomics 1998;14:241–339.

295. Endacott R, Clifford CM, Tripp JH. Can the needs of the critically ill child be identified usingscenarios? Experiences of a modified Delphi study. J Adv Nurs 1999;30:665–76.

296. Harrington JM. Research priorities in occupationalmedicine: a survey of United Kingdom medicalopinion by the Delphi technique. Occup EnvironMed 1993;51:289–94.

297. Charlton JRH, Patrick DL, Matthews G, West PA.Spending priorities in Kent: a Delphi study. J Epidemiol Community Health 1981;35:288–92.

298. Roberts H, Khee TS, Philip I. Setting priorities for measures of performance for geriatric medicalservices. Age Ageing 1994;23:154–7.

299. Hadorn DC, Holmes AC. The New Zealand priority criteria project. Part 1: Overview. BMJ 1997;314:131–4.


85

300. Gabbay J, Francis L. How much day surgery?Delphic predictions. BMJ 1988;297:1249–52.

301. Wilson DM, Kerr JR. An exploration of Canadiansocial values relevant to health care. Am J HealthBehav 1998;22:120–9.

302. Thomson WA, Ponder LD. Use of Delphimethodology to generate a survey instrument toidentify priorities for state allied health associations.Allied Health Behav Sci 1979;2:383–99.

303. Gallagher M, Bradshaw C, Nattress H. Policypriorities in diabetes care: a Delphi study. QualHealth Care 1996;5:3–8.

304. Burns TJ, Batavia AI, Smith QW, DeJong G. Primary health care needs of persons with physicaldisabilities: what are the research and servicepriorities? Arch Phys Med Rehabil 1990;71:138–43.

305. Kastein MR, Jacobs M, Roelof HvdH, Luttik K,Touw-Otten F. Delphi, the issue of reliability: aqualitative Delphi study in primary health care inthe Netherlands. Technological Forecasting Soc Change1993;44:315–23.

306. Festervand TA. An introduction and application offocus group research to the health care industry.Health Mark Q 1984;2(2–3):199–209.

307. Morgan DL. Focus groups as qualitative research.2nd ed. Thousand Oaks, CA: Sage; 1997.

308. Dowell J, Huby G, Smith C. Scottish ConsensusStatement on quality research in primary healthcare. Dundee: Tayside Centre for General Practice,University of Dundee; 1995.

309. Morgan D. Focus groups as qualitative research.Newbury Park, CA: Sage; 1988.

310. Krueger RA. Focus groups: a practical guide forapplied research. Newbury Park, CA: Sage; 1988.

311. Bowie C, Richardson A, Sykes W. Consulting thepublic about health service priorities. BMJ1995;311:1155–8.

312. Kuder LB, Roeder PW. Attitudes toward age-basedhealth care rationing: a qualitative assessment. J Aging Health 1995;7:301–27.

313. Cohen MB, Garrett KJ. Breaking the rules: a groupwork perspective on focus group research. Br J SocWork 1999;29:359–72.

314. Kitzinger J. The methodology of focus groups: theimportance of interaction between researchparticipants. Sociol Health Illn 1994;16:103–21.

315. Ward VM, Bertrand JT, Brown LF. Thecomparability of focus group and survey results:three case studies. Eval Rev 1991;15:266–83.

316. Carey MA, Smith MW. Enhancement of validitythrough qualitative approaches – incorporating the patients perspective. Eval Health Prof1992;15:107–14.

317. Powell RA, Single HM, Lloyd KR. Focus groups inmental health research: enhancing the validity ofuser and provider questionnaires. Int J Soc Psychiatry1996;42:193–206.

318. Wilkinson S. Focus groups in health research. J Health Psychol 1998;3:329–48.

319. Stevens PE. Focus groups: collecting aggregate-leveldata to understand community health phenomena.Public Health Nurs 1996;13:170–6.

320. Keller KL, Sliepcevich EM, Vitello EM, Lacey EP,Wright WR. Assessing beliefs about and needs ofsenior citizens using the focus group interview: aqualitative approach. Health Educ 1987;18:44–9.

321. Smith B, West P. Focus groups: giving voice to thecommunity. Perspectives 1998;22(3):2–7.

322. Bradley N, Sweeney K, Waterfield M. The health oftheir nation: how would citizens develop England’shealth strategy? Br J Gen Pract 1999;49:801–5.

323. Dolan P, Cookson R, Ferguson B. Effect ofdiscussion and deliberation on the public’s views of priority setting in health care: focus group study. BMJ 1999;318:916–19.

324. Ramirez AG, Shepperd J. The use of focus groupsin health research. Scand J Prim Health Care 1988;6 Suppl 1:S81–S90.

325. Weinberger M, Ferguson JA, Westmoreland G,Mamlin LA, Segar DS, Eckert GJ, et al. Can ratersconsistently evaluate the content of focus groups.Soc Sci Med 1998;46:929–33.

326. Southern DM, Batterham RW, Appleby NJ, Young D, Dunt D, Guibert R. The concept mapping method: an alternative to focus groupinquiry in general practice. Aust Fam Physician1999;28 Suppl 1:S35–S40.

327. Trochim WMK, Linton R. Conceptualization forplanning and evaluation. Eval Program Plann1986;9:289–308.

328. Lenaghan J. Involving the public in rationingdecisions. The experience of citizens juries. Health Policy 1999;49:45–61.

329. McIver S. Healthy debate? An independentevaluation of citizens’ juries in health settings.London: King’s Fund; 1998.

330. Renn O, Webler T, Rakel H, Dienel P, Johnson B.Public-participation in decision-making – a 3-stepprocedure. Policy Sci 1993;26:189–214.

331. Coote A, Lenaghan J. Citizens’ juries: theory intopractice. London: Institute for Public PolicyResearch; 1997.

332. Lenaghan J, New B, Mitchell E. Setting priorities: is there a role for citizens’ juries? BMJ1996;312:1591–3.

References

86

333. Davies S, Elizabeth S, Hanley S, New B, Sang B.Ordinary wisdom. London: King’s Fund; 1998.

334. Dunkerley D, Glasner P. Empowering the public?Citizens’ juries and the new genetic technologies.Crit Public Health 1998;8:181–92.

335. Reid N, Reid R, Morris D. Customer complaints in the National Health Service. J Nurs Manag1995;3:295–9.

336. Stronks K, Strijbis AM, Wendte JF. Who shoulddecide? Qualitative analysis of panel data frompublic, patients, healthcare professionals, andinsurers on priorities in health care. BMJ1997;315:92–6.

337. Rosenthal TC, Ferrara E, Hesler E. Providingbirthing services in rural health networks: copingwith change in New York State. J Rural Health1996;12:137–45.

338. Hirschfield RM, Keller MB, Panico S, Arons BS,Barlow D, Dandoff F, et al. The National Depressiveand Manic-Depressive Association ConsensusStatement on the Undertreatment of Depression.JAMA 1997;277:333–40.

339. Fernandez AM, Schrogie JJ, Wilson WW, Nash D.Technology assessment in healthcare: a review and description of a “best practice” technologyassessment process. Best Pract Benchmarking Healthc1997;2:240–53.

340. Coulter I, Adams A, Shekelle P. Impact of varyingpanel membership on ratings of appropriateness in consensus panels: a comparison of a multi- and single disciplinary panel. Health Serv Res1995;30:577–91.

341. Gundry KG, Heberlein TA. Do public meetingsrepresent the public? J Am Planning Assoc1984;50:175–82.

342. Broadbent B. Open to question. Health Serv J1998;108:30–1.

343. Gott M, Warren G. Neighbourhood health forums:local democracy at work. World Health Forum1991;12:413–18.

344. Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a researchtool for general practice? Fam Pract 1993;10:76–81.

345. DeBold B. The nominal group technique [Internet communication], 2000; http://www.radix.net/~ash2jam/TQM/nominal.htm

346. Rohrbaugh J. Improving the quality of groupjudgment: social judgment analysis and the nominalgroup technique. Organ Behav Hum Perform1981;28:272–88.

347. Hares T, Spencer J, Gallagher M, Bradshaw C, Webb I. Diabetes care: who are the experts? Qual Health Care 1992;1:219–24.

348. Jones J, Hunter D. Concensus methods for medicaland health services research. BMJ 1995;311:376–80.

349. Redman S, Carrick S, Cockburn J, Hirst S. Con-sulting about priorities for the NHMRC NationalBreast Cancer Centre: how good is the nominalgroup technique. Aust N Z J Public Health1997;21:250–6.

350. Streiner DL, Norman GR. Health MeasurementScales: a practical guide to their development anduse. Oxford: Oxford University Press; 1989.

351. Crombie IK, Davies.HTO. Research in health care:design, conduct and interpretation of healthservices research. Chichester: John Wiley; 1996.

352. Froberg DG, Kane RL. Methodology for measuringhealth-state preferences–II: scaling methods. J ClinEpidemiol 1989;42:459–71.

353. Phillips KA, Homan RK, Luft HS, Hiatt PH, Olson KR, Kearney TE, et al. Willingness to pay for poison control centers. J Health Econ1997;16:343–57.

354. Donaldson C. Developing the method of‘willingness to pay’ for assessment of communitypreferences for health care. Final report to Biomed2 Programme (PL950832) of the EuropeanCommission. Health Economics Research Unit,University of Aberdeen and Departments ofEconomics and Community Health Sciences,University of Calgary, 1999.

355. Donaldson C, Shackley P, Abdalla M. Usingwillingness to pay to value close substitutes: carrier screening for cystic fibrosis revisited. Health Econ 1997;6:145–59.

356. Chestnut L, Keller L, Lambert W, Rowe R.Measuring heart patients’ willingness to pay forchanges in angina symptoms. Med Decis Making1996;16:65–77.

357. San Miguel F, Ryan M, Scott A. Testing theassumptions of completeness and stability ofpreferences in discrete choice experiments. Paperpresented at the 2nd International Conference ofthe International Health Economics Association;1999 Jun; Rotterdam.

358. Shickle D. Public preferences for health care:prioritisation in the United Kingdom. Bioethics1997;11:277–90.

359. Gudex C, Dolan P, Kind P, Williams A. Health statevaluations from the general public using the visualanalogue scale. Qual Life Res 1996;5:521–31.

360. Dolan P, Kind P. Inconsistency and health statevaluations. Soc Sci Med 1996;42:609–15.

361. Llewellyn-Thomas H, Sutherland HJ, Tibshirani R,Ciampi A, Till JE, Boyd NF. The measurement ofpatients’ values in medicine. Med Decis Making1982;2:449–62.


87

362. Dolan P, Gudex C, Kind P, Williams A. The Time Trade-Off Method: results from a generalpopulation study. Health Econ 1996;5:141–54.

363. Ashby J, O’Hanlon M, Buxton MJ. The time trade-off technique: how do the valuations of breastcancer patients compare to those of other groups?Qual Life Res 1994;3:257–65.

364. Ubel PA, Loewenstein G, Scanlon D, Kamlet M.Individual utilities are inconsistent with rationingchoices: a partial explanation of why Oregon’s cost-effectiveness list failed. Med Decis Making1996;16:108–16.

365. Badia X, Roset M, Herdman M. Inconsistentresponses in three preference-elicitation methodsfor health states. Soc Sci Med 1999;49:943–50.

366. Bowling A. What people say about prioritisinghealth services. London: King’s Fund; 1999.

367. Frew E, Wolstenholme J, Whynes D. Willingness topay for colorectal cancer screening: faecal occultblood test versus flexible sigmoidoscopy. Paperpresented to the UK Health Economists’ StudyGroup Meeting; 1999 Jul; Aberdeen.

368. Miedzybrodzka Z, Semper J, Shackley P, Abdalla M,Donaldson C. Stepwise or couple antenatal carrierscreening for cystic fibrosis? Women’s preferencesand willingness to pay. J Med Genet 1995;32:282–3.

369. Ryan M, San Miguel F. Testing for consistency inwillingness to pay experiments. J Econ Psychol2000;21:305–17.

370. Babakus E, Mangold WG. Adapting the SERVQUALscale to hospital services: an empirical investigation.Health Serv Res 1992;26:767–86.

371. Murphy E, Dingwall R, Greatbatch D, Parker S,Watson P. Qualitative research methods in healthtechnology assessment: a review of the literature.Health Technol Assess 1998;2(14).

372. Mason J. Qualitative researching. London: Sage;1996.

373. Dingwall R, Murphy E, Watson P, Greatbatch D,Parker S. Catching goldfish: quality in qualitativeresearch. J Health Serv Res Policy 1998;3:167–72.

374. Lincoln YS, Guba E. Naturalistic enquiry. BeverlyHills, CA: Sage; 1985.

375. Seale C. The quality of qualitative research.London: Sage; 1999.

376. Kumar NC, Ganesh LS. A simulation-basedevaluation of the approximate and the exacteigenvector methods employed in AHP. Eur J Oper Res 1996;95:656–62.

377. Blaikie NWH. A critique of the use of triangulationin social research. Qual Quantity 1991;25:115–36.

378. Silverman D, editor. Qualitative research: theory,method and practice. London: Sage; 1997.

379. Walker R, editor. Applied qualitative research.Aldershot: Gower; 1985.

380. MacPherson IA, Williamson PJ. ‘Not quite what Imeant!’ – techniques of respondent validation. Res Policy Plann 1992;10:10–13.

381. Banister P, Burman E, Parker I, Taylor M, Tindall C.Qualitative methods in psychology: a researchguide. Buckingham: Open University Press; 1994.

382. Bryman A, Burgess R. Analyzing qualitative data.London: Routledge; 1994.

383. Conrad P, Reinharz S. Computers and qualitativedata. Editor’s introductory essay. Qual Sociol1984;7:1–2.

384. Glaser BG, Strauss AL. The discovery of groundedtheory: strategies for qualitative research. Chicago:Aldine Publishing Co.; 1967.

385. Strauss AL. Qualitative analysis for social scientists.Cambridge: Cambridge University Press; 1987.

386. Denzin NK, Lincoln YS, editors. Handbook ofqualitative research. Thousand Oaks, CA: Sage;1994.

387. Fink A. Evaluation fundamentals: guiding healthprograms, research, and policy. Newbury Park, CA:Sage; 1993.

388. Fitzpatrick R, Davey C, Buxton M, Jones DR.Evaluating patient-based outcome measures for usein clinical trials. Health Technol Assess 1988;2(14).

389. Kirk J, Miller M. Reliability and validity in quali-tative research. Newbury Park, CA: Sage; 1986.

390. O’Brien B, Gafni A. When do the “dollars” make sense? Toward a conceptual framework for contingent valuation studies in health care. Med Decis Making 1996;16:299.

391. Morrison GC, Gyldmark M. Appraising the use ofcontingent valuation. Health Econ 1992;1:233–43.

392. Arrow K, Solow R, Portney P, Leamer E, Radner R,Schuman H. Report of the NOAA panel of con-tingent valuation. Federal Register 1993;10:4601–14.

393. Drummond MF, Stoddart GL, Torrance GW.Methods for the economics evaluation of healthcare programmes. 2nd ed. Oxford: OxfordUniversity Press; 1997.

394. CASP’s qualitative checklist. 1999. URL:http://www.phru.org/casp/qualitative.html

395. Mays N, Pope C. Rigour and qualitative research.BMJ 1995;311:109–12.

396. Shackley P, Ryan M. Involving consumers in health care decision making. Health Care Anal1995;3:196–204.

References

88

397. Deleted.

398. Green PCF, Wind Y. Subjective evaluation models and conjoint measurement. Behav Sci1972;17:288–99.

399. Nord E. The validity of a visual analogue scale indetermining social utility weights for health states.Int J Health Plann Manage 1991;6:234–42.

400. Torrance GW. Social preferences for health states:an empirical evaluation of three measurementtechniques. Socioecon Plann Sci 1976;10:129–36.

401. Kaplan RM, Bush JW, Berry CC. Health statusindex: category rating versus magnitude estimationfor measuring levels of well-being. Med Care1979;17:501–25.

402. Sintonen H. An approach to measuring and valuinghealth states. Soc Sci Med 1981;15:55–65.

403. Richardson J. Cost utility analysis: what should bemeasured? Soc Sci Med 1994;39:7–21.

404. Johannesson M, Jonsson B, Karlsson G. Outcomemeasurement in economic evaluation. Health Econ1996;5:279–96.

405. Bleichrodt H, Johannesson M. An experimental test of a theoretical foundation for rating scalevaluation. Med Decis Making 1997;17:208–16.

406. Torrance GW. Utility approach to measuring health-related quality of life [review]. J Chron Dis1987;40:593–603.

407. Kaplan RM, Feeny D, Revicki DA. Methods forassessing relative importance in preference basedoutcome measures [review]. Qual Life Res1993;2:467–75.

408. Busschbach JJ, Horikx PE, van den Bosch JM,Brutel dlR, de Charro FT. Measuring the quality of life before and after bilateral lung transplant-ation in patients with cystic fibrosis. Chest1994;105:911–17.

409. Bakker C, Rutten M, van Doorslaer E, Bennett K,van der Linden S. Feasibility of utility assessment byrating scale and standard gamble in patients withankylosing spondylitis or fibromyalgia. J Rheumatol1994;21:269–74.

410. Sivertssen E, Fjeld NB, Abdelnoor M. Quality-of-lifeafter open-heart-surgery. Vasc Surg 1994;28:581–8.

411. Lenert LA, Morss S, Goldstein MK, Bergen MR,Faustman WO, Garber AM. Measurement of thevalidity of utility elicitations performed by com-puterized interview. Med Care 1997;35:915–20.

412. Revicki DA. Relationship between health utility and psychometric health status measures. Med Care1992;30 Suppl 5:MS274–82.

413. Morss SE, Lenert LA, Faustman WO. The sideeffects of antipsychotic drugs and patients’ qualityof life: patient education and preference assessmentwith computers and multimedia. Proceedings of theAnnual Symposium on Computer Applications inMedical Care; 1993. p. 17–21.

414. Van der Donk J, Levendag PC, Kuijpers AJ, RoestFHJ, Habbema JDF, Meeuwis CA, et al. Patientparticipation in clinical decision-making fortreatment of T3 laryngeal cancer: a comparison of state and process utilities. J Clin Oncol1995;13:2369–78.

415. Wolfson AD, Sinclair AJ, Bombardier C, McGreer A.Preference measurements for functional status instroke patients: interrater and intertechniquecomparisons. In: Kane RL, Kane RA, editors. Values and long term care. Lexington, MA:Lexington Books; 1982. p. 191–214.

416. Shiell A, King M, Briggs A. The consistency ofrating scale and time trade off techniques foreliciting preference weights for health states. Paper presented at the UK Health Economists’Study Group Meeting; 1993 Jul; Glasgow.

417. Krabbe PFM, EssinkBot ML, Bonsel GJ. Thecomparability and reliability of five health-statevaluation methods. Soc Sci Med 1997;45:1641–52.

418. O’Connor AM. Validation of a decisional conflictscale. Med Decis Making 1995;15:25–30.

419. O’Brien B, Viramontes JL. Willingness-to-pay – a valid and reliable measure of health statepreference. Med Decis Making 1994;14:289–97.

420. Gabriel SE, Campion ME, O’Fallon WM. Patientpreferences for nonsteroidal antiinflammatory drug related gastrointestinal complications andtheir prophylaxis. J Rheumatol 1993;20:358–61.

421. Sutherland HJ, Lockwood GA, Minkin S, Tritchler DL, Till JE, Llewellyn-Thomas HA.Measuring satisfaction with health care: a com-parison of single with paired rating strategies. Soc Sci Med 1989;28:53–8.

422. Rutten-van Molken MP, Bakker CH, van DoorslaerEK, van der Linden S. Methodological issues ofpatient utility measurement. Experience from two clinical trials. Med Care 1995;33:922–37.

423. Pinto PJ. Is the person trade-off a valid method for allocating health care resources? Health Econ1997;6:71–81.

424. O’Leary JF, Fairclough DL, Jankowski MK, Weeks JC. Comparison of time-tradeoff utilities and rating scale values of cancer patients and their relatives: evidence for a possible plateaurelationship. Med Decis Making 1995;15:132–7.

425. Parducci A. Contextual effects. A range–frequencyanalysis. In: Cartarette E, Friedman M, editors.Handbook of perception. Vol II. New York:Academic Press; 1974. p. 127–41.


89

426. Dyer JS, Sarin RK. Relative risk-aversion. Manag Sci1982;28:875–86.

427. Anderson N. Functional measurement and psycho-physical judgement. Psychol Rev 1970;77:153–70.

428. Anderson N. Integration theory and attitudechange. Psychol Rev 1971;78:171–206.

429. Appels A, Bosma H, Grabauskas V, Gostautas A,Sturmans F. Self-rated health and mortality in aLithuanian and a Dutch population. Soc Sci Med1996;42:681–9.

430. Adelman RD, Fields SD, Jutagir R. Geriatriceducation. 2. The effect of a well elderly programon medical-student attitudes toward geriatric-patients. J Am Geriatr Soc 1992;40:970–3.

431. Avis M. Incorporating patients’ voices in the auditprocess. Qual Health Care 1997;6:86–91.

432. Carr-Hill R. The measurement of patientsatisfaction. J Public Health Med 1992;14:236–49.

433. Cleary PD. The importance of patient surveys. BMJ 1999;319:720–1.

434. Locker D, Dunt D. Theoretical and methodologicalissues in sociological studies of consumer satisfac-tion with medical care. Soc Sci Med 1978;12:283–92.

435. Williams B. Patient satisfaction: a valid concept? Soc Sci Med 1994;38:509–16.

436. McKinley RK, Manku-Scott T, Hastings AM, French DP, Baker R. Reliability and validity of a new measure of patient satisfaction with out of hours primary medical care in the UK:development of a patient questionnaire. BMJ 1997;314:193–8.

437. Cohen G, Forbes J, Garraway M. Can differentpatient satisfaction survey methods yield consistentresults? Comparison of three surveys. BMJ1996;313:841–4.

438. Avis M. Incorporating patients’ voices in the auditprocess. Qual Health Care 1997;6:86–91.

439. Porter M, MacIntyre S. What is, must be best: a research note on conservative or differentialresponses to antenatal care provision. Soc Sci Med1984;19:1197–200.

440. Fitzpatrick R, Hopkins A. Problems in theconceptual framework of patient satisfactionresearch. Sociol Health Illn 1983;5:297–311.

441. Cleary PD, McNeill BJ. Patient satisfaction as anindicator of quality care. Inquiry 1998;25:25–36.

442. Oswald SL, Turner DE, Snipes RL, Butler D. Qualitydeterminants and hospital satisfaction. Perceptionsof the facility and staff might be key influencingfactors. Mark Health Serv 1998;18:18–22.

443. Mishra DP, Singh J, Wood V. An empiricalinvestigation of two competing models of patientsatisfaction. J Ambul Care Mark 1991;4(2):17–36.

444. Shewchuk RM, O’Connor SJ, White JB. In search ofservice quality measures: some questions regardingpsychometric properties. Health Serv Manage Res1991;4:65–75.

445. Tomes A, Ng S. Service quality in hospital care: the development of an in-patient questionnaire. Int J Health Care Qual Assur 1995;8(3):25–33.

446. Youssef F, Nel D, Bovaird T. Service quality in NHShospitals. J Manag Med 1995;9:66–74.

447. Hart MC. Measuring perception of quality in NHS clinics using SERVQUAL methodology. In: Richards B, editor. Current perspectives inhealthcare computing. Weybridge, Surrey: BJHC;1996. p. 37–42.

448. Dyck D. Gap analysis of health services. Client satisfaction surveys [review]. AAOHN J1996;44(11):541–9.

449. Ryan M. Using consumer preferences in health care decision-making: the application of conjointanalysis. London: Office of Health Economics;1997.

450. McFadden D. Conditional logit analysis ofqualitative choice behavior. Berkeley, CA: University of California at Berkeley; 1973.

451. Hannemann W. Welfare evaluations in contingentvaluation experiments with discrete choices: reply.Am J Agric Econ 1984;69:332–41.

452. Peralta-Carcelen MFC, Coston D, Dolan JG.Preferences of pregnant women and physicians fortwo strategies for prevention of early onset group Bstreptococcal sepsis in neonates. Arch Pediatr AdolescMed 1997;151:712–18.

453. Dolan JG, Bordley DR. Individualized patientdecision making using the analytic hierarchyprocess (AHP): reliability, validity, and clinicalusefulness [meeting abstract]. Med Decis Making1991;11:322.

454. Dolan JG, Bordley DR. Bedside decision makingusing the analytic hierarchy process. Med DecisMaking 1992;12:344.

455. Dolan JG. The role of diagnostic endoscopy in the management of low risk patients with acutepyelonephritis [meeting abstract]. Clin Res1992;40:580A.

456. Blumenschein K, Johannesson M. Relationshipsbetween quality of life instruments, health stateutilities, and willingness to pay in patients withasthma. Annal Allergy Asthma Immunol1998;80:189–94.

457. Torrance GW, Feeny DH, Furlong WJ, Barr RD,Zhang Y, Wang Q. Multiattribute utility function for a comprehensive health status classificationsystem – Health Utilities Index Mark 2. Med Care1996;32:702–22.

References

90

458. Shackley P, Cairns J. Evaluating the benefits ofantenatal screening: an alternative approach. Health Policy 1996;36:103–15.

459. Gage BF, Cardinalli AB, Owens DK. The effect of stroke and stroke prophylaxis with aspirin orwarfarin on quality of life. Arch Intern Med1996;156:1829–36.

460. Rabin R, Rosser RM, Butler C. Impact of diagnosison utilities assigned to states of illness. J R Soc Med1993;86:444–8.

461. Ramsey SD, Patrick DL, Lewis S, Albert RK, Raghu G. Improvement in quality of life after lung transplantation: a preliminary study. TheUniversity of Washington Medical Center LungTransplant Study Group. J Heart Lung Transplant1995;14:870–7.

462. Dolan P, Gudex C, Kind P, Williams A. Valuinghealth states: a comparison of methods. J HealthEcon 1996;15:209–31.

463. Patrick DL, Starks HE, Cain KC, Uhlmann RF,Pearlman RA. Measuring preferences for healthstates worse than death. Med Decis Making1994;14:9–18.

464. Hall J, Gerard K, Salkeld G, Richardson J. A costutiliy analysis of mammography screening inAustralia. Soc Sci Med 1992;34:993–1004.

465. Thompson MS. Willingness to pay and accept risks to cure chronic disease. Am J Public Health1986;76:392–6.

466. Reed WW, Herbers JEJ, Noel GL. Cholesterol-lowering therapy: what patients expect in return. J Gen Intern Med 1993;8:591–96.

467. Torrance GW. Measurement of health state utilitiesfor economic appraisal [review]. J Health Econ1986;5:1–30.

468. Bosch JL, Hunink MG. The relationship betweendescriptive and valuational quality-of-life measuresin patients with intermittent claudication [review].Med Decis Making 1996;16:217–25.

469. Zug KA, Littenberg B, Baughman RD, Kneeland T,Nease RF, Sumner W, et al. Assessing the prefer-ences of patients with psoriasis. A quantitative,utility approach. Arch Dermatol 1995;131:561–8.

470. Bala MV, Wood LL, Zarkin GA, Norton EC, Gafni A,O’Brien B. Valuing outcomes in health care: acomparison of willingness to pay and quality-adjusted life years. J Clin Epidemiol 1998;51:667–76.

471. Scott A. Giving things up to have more of others.The implications of limited substitutability foreliciting preferences in health and health care.Aberdeen: Health Economics Research Unit; 1998.HERU Discussion Papers 01/98.

472. Von Winterfeldt D. Additivity and expected utilityin risky multiattribute preferences. J Math Psychol1980;21:66–82.

473. Tversky A, Sattath S, Slovic P. Contingent weightingin the judgement of choice. Psychol Rev1988;95:371–84.

474. Slovic P, Tversky A. Who accepts Savage’s axiom?Behav Sci 1974;19:368–73.

475. Kahneman D, Tversky A. Prospect theory: ananalysis of decision making under risk. Econometrica1979;47:263–92.

476. Bell DE. Regret in decision making underuncertainty. Oper Res 1982;30:961–81.

477. Loomes G, Sugden R. Regret theory: an alternativetheory of rational choice under uncertainty. Econ J1982;92:805–24.

478. Slovic P, Lichtenstein S. Preference reversals: abroader perspective. Am Econ Rev 1983;73:596–605.

479. Loomes G, Sugden R. The importance of whatmight have been. In: Hagen O, Wenstop F, editors.Progress in utility and risk theory. Dordrecht:Reidel; 1984. p. 219–35.

480. Bell DE. Disappointment in decision making underuncertainty. Oper Res 1985;33:1–27.

481. Loomes G, Sugden R. Disappointment and dynamicconsistency in choice under uncertainty. Rev EconStud 1986;53:271–82.

482. Ryan M. Valuing psychological factors in theprovision of assisted reproductive techniques usingthe economic instrument of willingness to pay. J Econ Psychol 1998;19:179–204.

483. Smith RD. Is regret theory an alternative basis forestimating the value of healthcare interventions?Health Policy 2000;37:105–15.

484. Fryback DG, Dasbach EJ, Klein R, Klein BE, Dorn N, Peterson K, et al. The Beaver Dam HealthOutcomes Study: initial catalog of health-statequality factors. Med Decis Making 1993;13:89–102.

485. Johnson ES, Sullivan SD, Mozaffari E, Langley PC,Bodsworth NJ. A utility assessment of oral andintravenous ganciclovir for the maintenancetreatment of AIDS-related cytomegalovirus retinitis.Pharmacoeconomics 1996;10:623–9.

486. Krumins PE, Fihn SD, Kent DL. Symptom severityand patients’ values in the decision to perform atransurethral resection of the prostate. Med DecisMaking 1988;8:1–8.

487. Detsky AS, McLaughlin JR, Abrams HB, L’Abbe KA,Whitwell J, Bombardier C, et al. Quality of life ofpatients on long-term total parenteral nutrition athome. J Gen Intern Med 1986;1:26–33.

488. Glasziou PP, Bromwich S, Simes RJ. Quality of lifesix months after myocardial infarction treated withthrombolytic therapy. AUS-TASK Group. Australianarm of International tPA/SK Mortality Trial. Med JoAust 1994;161:532–6.


91

489. Kreibich DN, Vaz M, Bourne RB, Rorabeck CH,Kim P, Hardie R, et al. What is the best way ofassessing outcome after total knee replacement?Clin Orthop 1996;331:221–5.

490. Handler RM, Hynes LM, Nease RFJ. Effect of locus of control and consideration of future con-sequences on time tradeoff utilities for currenthealth. Qual Life Res 1997;6:54–60.

491. Churchill DN, Torrance GW, Taylor DW.Measurement of quality of life in end-stage renaldisease: the time trade-off approach. Clin Invest Med1987;10(1):14–20.

492. Molzahn AE, Northcott HC, Hayduk L. Quality oflife of patients with end stage renal disease: astructural equation model. Qual Life Res1996;5:426–32.

493. Kemp S. Magnitude estimation of the utility ofpublic-goods. J Appl Psychol 1991;76:533–40.

494. Robinson A, Dolan P, Williams A. Valuing healthstatus using VAS and TTO: what lies behind thenumbers? Soc Sci Med 1997;45:1289–97.

495. Zethraeus N, Johannesson M, Henriksson P, Strand RT. The impact of hormone replacementtherapy on quality of life and willingness to pay. Br J Obstet Gynaecol 1997;104:1191–5.

496. Swan JS, Fryback DG, Lawrence WF, Katz DA,Heisey DM, Hagenauer ME, et al. MR and con-ventional angiography: work in progress towardsassessing utility in radiology. Acad Radiol1999;4:475–82.

497. Sackett DL, Torrance GW. The utility of differenthealth states as perceived by the general public. J Chronic Dis 1978;31:697–704.

498. Lipscomb J. Value preferences for health: meaning,measurement, and use in program evaluation. In:Kane RL, Kane RA, editors. Values and long termcare. Lexington, MA: Lexington Books; 1982. p. 27–83.

499. Sutherland HJ, Llewellyn-Thomas H, Boyd NF, Till JE. Attitudes toward quality of survival. Theconcept of “maximal endurable time”. Med DecisMaking 1982;2:299–309.

500. Wakker P, Stigglebout A. Explaining distortions inutility elicitation through the rank dependentmodel for risky choices. Med Decis Making1995;15:180–5.

501. Irvine EJ. Quality of Life in inflammatory boweldisease: biases and other factors affecting scores.Scand J Gastroenterol Suppl 1995;208:136–40.

502. Dolan P, Gudex C. Time preference, duration andhealth state valuations. Health Econ 1995;4:289–99.

503. Green C, Brazier J, Deverill M. A review of theevidence surrounding health state valuationtechniques (VAS, TTO, SG, PTO, ME). Paperpresented at the UK Health Economists’ StudyGroup Meeting; 1998 Jul; Galway.

504. Nord E, Richardson J, Macarounas-Kirchmann K.Social evaluation of health care versus personalevaluation of health states. Evidence on the validityof four health-state scaling instruments usingNorwegian and Australian surveys. Int J TechnolAssess Health Care 1993;9:463–78.

505. Gafni A. Willingness to pay in the context of aneconomic evaluation of healthcare programs:theory and practice. Am J Manag Care 1997;3 Suppl:S21–S32.

506. Pauly MV. Valuing health care benefits in moneyterms. In: Sloan FA, editor. Valuing health care:costs, benefits, and effectiveness of pharmaceuticalsand other medical technologies. Cambridge:Cambridge University Press; 1995. p. 99–124.

507. Donaldson C, Farrar S, Mapp T, Walker A, Macphee S. Assessing community values in healthcare: is the ‘willingness to pay’ method feasible?Health Care Anal 1997;5:7–29.

508. Thompson MS, Read JL, Liang M. Feasibility ofwillingness-to-pay measurement in chronic arthritis.Med Decis Making 1984;4:195–215.

509. Kartman B, Stalhammar NO, Johannesson M.Contingent valuation with an open-ended follow-upquestion: a test of scope effects. Health Econ1997;6:637–9.

510. Thompson MS, Read JL, Laing M. Willingness to pay concepts for social decisions in health. In:Kane RL, Kane RA, editors. Values and long termcare. Lexington, MA. Lexington Books; 1982. p. 103–26.

511. Donaldson C, Hundley V, Mapp T. Willingness topay: a method for measuring preferences formaternity care? Birth 1998;25:32–9.

512. Donaldson C, Thomas R, Torgerson DJ. Validity of open-ended and payment scale approaches toeliciting willingness to pay. Appl Econ 1997;29:79–84.

513. Johannesson M, Johansson P, Kristrom B, Borgquist L, Jönsson B. Willingness to pay for lipid lowering: a health production functionapproach. Appl Econ 1993;25:1023–31.

514. Eckerlund I, Johannesson M, Johansson P-O,Tambour M, Zethraeus N. Value for money? Acontingent valuation study of the optimal size of the Swedish health care budget. Health Policy1999;34:135–43.

515. Johannesson M, O’Conor RM, Kobelt-Nguyen G,Mattiasson A. Willingness to pay for reducedincontinence symptoms. Br J Urol 1997;80:557–62.

References

92

516. Ryan M. Using willingness to pay to assess thebenefits of assisted reproductive techniques. HealthEcon 1996;5:543–58.

517. Ryan M. Should government fund assistedreproductive techniques? A study using willingnessto pay. Appl Econ 1997;29:841–9.

518. Lindholm L, Rosén ME, Stenbeck ME.Determinants of willingness to pay taxes for a community-based prevention programme. Scand J Soc Med 1997;25:126–35.

519. Flowers CR, Garber AM, Bergen MR, Lenert LA.Willingness-to-pay utility assessment: feasibility ofuse in normative patient decision support systems. J Am Med Informatic Assoc Proc 1997;223–7.

520. Coley CM, Li Y-H, Medsger AR, Marrie TJ, Fine MJ,Kapoor WN, et al. Preferences for home vs. hospitalcare among low-risk patients with community-acquired pneumonia. Arch Intern Med1996;156:1565–71.

521. Dranitsaris G. A pilot study to evaluate the feasi-bility of using willingness to pay as a measure ofvalue in cancer supportive care: an assessment ofamistofine cytoprotection. Support Care Cancer1997;5:489–99.

522. Lee SJ, Neumann PJ, Churchill WH, Cannon ME,Weinstein MC, Johannesson M. Patients’ willingnessto pay for autologous blood donation. Health Policy1997;40:1–12.

523. Johannesson M, Åberg H, Agréus L, Borgquist L,Jönsson B. Cost–benefit analysis of non-pharmacological treatment of hypertension. J Intern Med 1991;230:307–12.

524. Johannesson M. Economic evaluation of lipidlowering – a feasibility test of the contingentvaluation approach. Health Policy 1992;20:309–20.

525. Van der Pol M, Cairns J. Eliciting individual timepreferences for own health using a dichotomouschoice question with follow-up. Paper presented tothe 4th Nordic Health Econometrics Workshop;1998 Aug 20; Oslo.

526. Schkade DA, Payne JW. How people respond tocontingent valuation questions: a verbal protocolanalysis of willingness to pay for an environmentalregulation. J Environ Econ Manage 1994;26:88–109.

527. Loehman ET, Berg SV, Arroyo AA, Hedinger RA,Schwartz JM, Shaw ME, et al. Distributional analysisof regional benefits and cost of air quality control. J Environ Econ Manage 1979;6:222–43.

528. Pennie RA, O’Connor A, Garvock M, Drake E.Factors influencing the acceptance of Hepatitis Bvaccine by students in health disciplines in Ottawa.Can J Public Health 1991;82:12–15.

529. Asenso-Okyere WK, Osei-Akoto I, Anum A, Appiah EN. Willingness to pay for health insurancein a developing country – a pilot study of theinformal sector of Ghana using contingentvaluation. Health Policy 1997;42:223–37.

530. Appel L, Steinberg E, Powe N, Anderson G, Dwyer S, Faden R. Risk reduction from low osmality contrast media: what do patients think it’s worth? Med Care 1990;28:324–37.

531. Berwick DM, Weinstein MC. What do patientsvalue? Willingness to pay for ultrasound in normalpregnancy. Med Care 1985;23:881–93.

532. Anderson G, Black C, Dunn E, Alonso J, Christian-Norrega J, Folmer-Anderson T, et al.Willingness to pay to shorten waiting time forcataract surgery. Health Aff 1997;16:181–91.

533. Acton J. Evaluating public programmes to save lives:the case of heart attacks. Santa Monica, CA: RANDCorporation; 1973. Report no. R950RC.

534. O’Brien B, Goeree R, Gafni A, Torrance GW, Pauly MV, Erder H, et al. Assessing the value of a new pharmaceutical: a feasibility study ofcontingent valuation in managed care. Med Care1998;36:370–84.

535. Ramsey SD, Sullivan SD, Psaty BM, Patrick DL.Willingness to pay for antihyperintensive care:evidence from a staff-model HMO. Soc Sci Med1997;44:1911–17.

536. Gore P, Madhavan S. Consumers’ preference andwillingness to pay for pharmacist counselling fornon-prescription medicines. J Clin Pharm Ther1994;19:12–25.

537. Johannesson M, Johansson B. On the value ofchanges in life-expectancy: blips versus parametricchanges. J Risk Uncertainty 1997;15:221–39.

538. O’Brien B, Novosel S, Torrance G, Streiner D.Assessing the economic value of a newantidepressant: a willingness to pay approach.Pharmacoeconomics 1995;8:34–45.

539. Osmond M, Klassen T, Quinn J. Economic com-parison of a tissue adhesive and suturing in therepair of pediatric facial lacerations. J Pediatr1995;126:892–5.

540. Reutzel TJ, Furmaga E. Willingness to pay forpharmacist services in a veterans administrationhospital. J Res Pharm Econ 1993;5(2):89–114.

541. Burgoyne CB. Distributive justice and rationing in the NHS: framing effects in press coverage of acontroversial decision. J Community Appl Soc Psychol1997;7:119–36.

542. Fox-Rushby J. Willingness to pay as a method for valuing health related quality of life. Paperpresented at the UK Health Economists’ StudyGroup Meeting; 1991 Jul; Aberdeen.


93

543. Granberg M, Wikland M, Nilsson L, Hamberger L.Couples’ willingness to pay for IVF/ET. Acta ObstetGynecol Scand 1995;74:199–202.

544. Walraven G. Willingness to pay for district hospitalsin rural Tanzania. Health Policy Plan 1996;11:428–37.

545. Fischer GW. Willingness to pay for probibalisticimprovements in health status: a psychologicalperspective. In: Mushkin SJ, Dunlop DW, editors.Health: what is it worth? New York: Pergamon;1979. p. 167–200.

546. Garbacz C, Thayer M. An experiment in valuingsenior companion program services. J Hum Resour1983;18:147–53.

547. Kobelt G. Economic considerations and outcomemeasurement in urge incontinence. Urology1997;50(6A):100–7.

548. Muller A, Reutzel TJ. Willingness to pay forreduction in fatality risk: an exploratory survey. Am J Public Health 1984;74:808–12.

549. Baron J. Confusion of relative and absolute risk invaluation. J Risk Uncertainty 1997;14:301–9.

550. Gyldmark M. Preferences for health care services in Denmark. An investigation through contingentvaluation. Paper presented at the 14th Meeting ofthe Nordic Health Economists Group; 1993 Aug18–20; Tromso.

551. Olsen JA. Aiding priority setting in health care: isthere a role for the contingent valuation method?[review]. Health Econ 1997;6:603–12.

552. Kartman B, Andersson F, Johannesson M.Willingness to pay for reductions in angina pectorisattacks. Med Decis Making 1996;16:248–53.

553. Kartman B, Stalhammar NO, Johannesson M.Valuation of health changes with the contingentvaluation method: a test of scope and questionorder effects. Health Econ 1996;5:531–41.

554. Viscusi WK, Magat WA, Huber J. Pricing environ-mental health risks: survey assessments of risk–riskand risk–dollar trade-offs for chronic bronchitis. J Environ Econ Manage 1991;21:32–51.

555. Ryan M, Scott DA, Donaldson C. Valuing healthcare using willingness to pay: a comparison of thepayment card and dichotomous choice methods.Paper presented to the 2nd International Confer-ence of the International Health EconomicsAssociation; 1999 Jun 7–9; Rotterdam.

556. Johannesson M. Economic evaluation of hyper-tension treatment. Int J Technol Assess Health Care1992;8:506–23.

557. Lee SJ, Liljas B, Neumann PJ, Weinstein MC,Johannesson M. The impact of risk information onpatients’ willingness to pay for autologous blooddonation. Med Care 1998;36:1162–73.

558. Stalhammar NO. An empirical note on willingnessto pay and starting-point bias. Med Decis Making1996;16:242–7.

559. Ryan M, Scott DA, Donaldson C. Econometricissues raised in the analysis of payment scale andclosed-ended contingent valuation data sets inhealth care. Paper presented to the 5th NordicHealth Econometrics Workshop; 1998 Aug 20; Oslo.

560. Ryan M, Ratcliffe J. Some issues in the applicationof closed-ended willingness to pay studies to valuinghealth goods: an application to antenatal care inScotland. Appl Econ 2000;32:643–51.

561. Mullen PM. Public involvement in health-carepriority setting: an overview of methods for elicitingvalues. Health Expectations 1999;2:222–34.

562. Srivastava J, Connolly T, Beach LR. Do rankssuffice? A comparison of alternative weightingapproaches in value elicitation. Organ Behav HumDecis Process 1995;63:112–16.

563. Mullen P, Spurgeon P. Priority setting and thepublic. Abingdon, Oxon: Radcliffe Medical Press;2000.

564. Hoinville G, Courtenay G. Measuring consumerpriorities. In: O’Riordan T, D’Arge RC, editors.Progress in resource management and environ-mental planning. Vol. 1. Chichester: Wiley; 1979. p. 143–70.

565. Browne JP, McGee HM, O’Boyle CA. Conceptualapproaches to the assessment of quality of life.Psychol Health 1997;12:737–51.

566. Bowling A, Jacobson B, Southgate L. Health-servicepriorities – explorations in consultation of thepublic and health-professionals on priority settingin an inner London health district. Soc Sci Med1993;37:851–7.

567. Clarke R. Developments in deliberative approaches.Paper presented at a seminar, ‘New methods forinvolving the public in decisions’; 1999 Sep;Birmingham.

568. Holloway I. Basic concepts for qualitative research.Oxford: Blackwell Science; 1997.

569. The Penguin dictionary of sociology. 2nd ed.Harmondsworth: Penguin Books; 1988.

570. Murphy MK, Black NA, Lamping DL, McKee CM,Sanderson CFB, Askham J, et al. Concensusdevelopment methods, and their use in clinicalguideline development. Health Technol Assess1998;2(3).

571. Williams PL, Webb C. The Delphi technique: a methodological discussion. J Adv Nurs1994;19:180–6.

572. Donovan J, Coast J. Public preferences in prioritysetting – unresolved issues. In: Malek M, editor.Setting priorities in health care. Chichester: JohnWiley & Sons, 1994; p. 31–45.

References

94

573. Hemenway D, Killen A. Complainers andnoncomplainers. J Ambulatory Care Manage1989;12(3):19–27.

574. Kitzhaber JA. Prioritising health services in an era of limits: the Oregon experience. BMJ1993;307:377.

575. Dean H, Gale K, Woods R. ‘This isn’t very typicalI’m afraid’: observing community care complaintsprocedures. Health Soc Care Community1996;4:338–46.

576. Eriksson CG. Focus groups and other methods forincreased effectiveness of community intervention –a review [review]. Scand J Primary Health Care Suppl1988;1:73–80.

577. Beaudin CL, Pelletier LR. Consumer-basedresearch: using focus groups as a method forevaluating quality of care. J Nurs Care Qual1996;10(3):28–33.

578. Tang KC, Davis A. Critical factors in the deter-mination of focus group size [review]. Fam Pract1995;12:474–5.

579. Birdwell SW, Calesaric H. Identifying health careneeds of rural Ohio citizens: an evaluation of a two-stage methodology. J Rural Health 1996;12:130–6.

580. Merton GB, Levine RJ, Koocher GP, Rosenthal R,Thompson WC. Community consultation in sociallysensitive research: lessons from clinical trials fortreatment of AIDS. Am Psychol 1988;43:573–81.

581. Price D. Choices without reasons: citizens’ juriesand policy evaluation. J Med Ethics 2000;26:272–6.

582. Seiler H. Review of “Planning Cells:” problems oflegitimation. In: Renn O, Webler T, Wiedemann P,editors. Fairness and competition in citizenparticipation: evaluating models for environmentaldiscourse. Dordrecht: Kluwer Academic; 1995. p. 144–51.

583. Lupton C, Peckham S, Taylor P. Managing publicinvolvement in healthcare purchasing.Buckingham: Open University Press; 1998.

584. Chess C, Purcell K. Public participation and theenvironment: do we know what works? Environ SciTechnol 1999;33:2685–92.

585. Jessop EG. Public meetings deserve proper science.J Public Health Med 1999;21:365–6.

586. Kocur G, Alder T, Hyman W, Aunet B. Guide to forecasting travel demand with direct utilityassessment: Final report to the US Department of Transportation. Hanover, NH: Resource PolicyCenter, Thayer School of Engineering, DartmouthCollege; 1982.

587. Cookson R. Incorporating psychosocial consider-ations into health valuation: an experimental study.J Health Econ 2000;19:369–401.

588. Majid I, Sinden JA, Randall A. Benefit valuation ofincrements to existing systems of public facilities.Land Econ 1983;59:377–92.

589. Hoehn J. Valuing the multidimensional impacts of environmental policy: theory and methods. Am Agric Assoc 1991;73:289–99.

590. Hoehn J, Randall A. Too many proposals pass thebenefit–cost test. Am Econ Rev 1989;79:544–51.

591. Gafni A. Willingness-to-pay as a measure of benefits:relevant questions in the context of public decisionmaking about health care programmes. Med Care1991;29:1246–52.

592. Benbassat J, Pilpel D, Tidhar M. Patients’ prefer-ences for participation in clinical decision making:a review of published studies. Behav Med1998;24:81–8.

593. Guadagnoli E, Ward P. Patient participation indecision-making. Soc Sci Med 1998;47:329–39.

594. Heginbotham C. Health care priority setting: asurvey of dcotors, managers, and the generalpublic. In: Smith R, editor. Rationing in action.London: BMJ Publishing; 1993. p. 141–56.

595. Bowling A. Health care rationing: the public’sdebate. BMJ 1996;312:670–4.

596. Kneeshaw J. What does the public think aboutrationing? A review of the evidence. In: New B,editor. Rationing: talk and action in health care.London: King’s Fund/BMJ Publishing Group; 1997. p. 58–76.

597. Coast J. Rationing within the NHS should beexplicit: the case against. BMJ 2000;314:1118–22.

598. Coast J. Explicit rationing, deprivation disutility anddenial disutility: evidence from a qualitative study.In: Coulter A, Ham C, editors. The global challengeof health care rationing. Buckingham: OpenUniversity Press; 2000. p. 192–200.

599. Fischhoff B. Value elicitation: is there anything inthere? Am Psychol 1991;46:835–47.

600. Gregory R, Lichestein S, Slovic P. Valuing environ-mental resources: a constructive approach. J RiskUncertainty 1993;7:177–97.

601. Slovic P. The construction of preference. Am Psychol1995;50:364–71.

602. Shiell A, Hawe P, Seymour J. Values and preferences are not necessarily the same. Health Econ 1997;6:515–18.

603. Snowdon H, Jones G. Local advisory groups in the Sedgefield area. Paper presented at a seminar,‘New methods for involving the public in decisions’;1999 Sep; Birmingham.

604. Cartwright A. The dignity of labour: a study ofchildbearing and induction. London: Tavistock;1979.


95

605. Bate A, Ryan M. Examining patient preferences for junior doctors versus specialist nurses in theprovision of rheumatology services: a discretechoice experiment. Aberdeen: University ofAberdeen; 1998. HERU Discussion Paper 05/98.

606. Salkeld G, Ryan M, Short L. The veil of experience:do consumers prefer what they know best? HealthEcon Lett 2000;4:4–9.

607. Thayler R. Toward a positive theory of consumerchoice. J Behav Organ 1980;1:39–60.

608. Samuelson W, Zeckhauser R. Status quo bias indecision making. J Risk Uncertainty 1988;1:7–59.

609. Knetsch JL, Sinden JA. Wilingness to pay andcompensation demanded: experimental evidence of an unexpected disparity in measures of value. Q J Econ 1984;99:507–21.

610. Knetsch JL, Sinden JA. The persistence ofevaluation disparities. Q J Econ 1987;102:691–5.

611. Knetsch JL. The endowment effect and evidence of non-reversible indifference curves. Am Econ Rev1989;790:1277–84.

612. Morrison GC. Willingness to pay and willingness to accept: some evidence of an endowment effect.Appl Econ 1997;29:411–17.

613. Morrison GC. Resolving differences in willingnessto pay and willingness to accept: comment. Am EconRev 1997;87:236–40.

614. Klein L. “Planning for Real” and other methods ofpublic participation. Paper presented at a seminar,‘New methods for involving the public in decisions’;1999 Sep; Birmingham.

615. Tashhakkori A, Teddlie C. Mixed methodology:combining qualitative and quantitative approaches.London: Sage; 1998.

616. Ferraz MB, Quaresma MR, Goldsmith CH, Bennett K, Atra E. Corticosteroids in patients with rheumatoid-arthritis – utility. Rev Rhum1994;61:255–9.

617. Read JL, Quinn RJ, Berwick DM, Fineberg HV,Weinstein MC. Preferences for health outcomes.Comparison of assessment methods. Med DecisMaking 1984;4:315–29.

618. Clarke AE, Goldstein MK, Michelson D, Garber AM,Lenert LA. The effect of assessment method andrespondent population on utilities elicited forGaucher disease. Qual Life Res 1997;6:169–84.

619. Freeman JK, Szeinbach SL, Barnes JH, Garner DD,Gilbert FW. Assessing the need for student healthservices using maximum difference conjointanalysis. J Res Pharm Econ 1998;9(3):35–49.

620. Szeinbach SL, Barnes JH, McGhan WF, Murawski MM, Corey R. Using conjoint analysis toevaluate health state preferences. Drug Inf J1999;33:849–58.

621. McNeil BJ, Pauker SG, Sox HCJ, Tvershy A. On theelicitation of preferences for alternative therapies.N Engl J Med 1982;306:1259–62.

622. Eckman MH. A counterpoint to the analytichierarchy process. Med Decis Making 1989;9:57–8.

623. Loomes G, McKenzie L. The use of QALY’s inhealth care decision making. Soc Sci Med1989;28:299–308.

624. Laupacis A, Bourne R, Rorabeck C, Feeny D, Wong C, Tugwell P, et al. The effect of elective totalhip replacement on health-related quality of life. J Bone Joint Surg Am 1993;75:1619–26.

625. Buckingham JK, Birdsall J, Douglas JG. Comparingthree versions of the time tradeoff: time for achange? Med Decis Making 1996;16:335–47.

626. Mehrez A, Gafni A. Healthy-years equivalents versusquality-adjusted life years: in pursuit of progress.Med Decis Making 1993;13:287–92.

627. Redelmeier DA, Heller DN. Time preference inmedical decision making and cost-effectivenessanalysis. Med Decis Making 1993;13:212–17.

628. Nord E. The trade-off between severity of illnessand treatment effect in cost–value analysis of health care. Health Policy 1993;24:227–38.

629. Donaldson C, Shackley P, Abdalla M,Miedzybrodzka Z. Willingness to pay for antenatal carrier screening for cystic fibrosis. Health Econ 1995;4:439–52.

630. Easthaugh S. Valuation of the benefits of risk-freeblood. Int J Technol Assess Health Care 1991;7:51–7.

631. Mills A, Fox-Rushby J, Aikins M, D’Alessandro U,Cham K, Greenwood B. Financing mechanisms for village activities in The Gamba and their impli-cations for financing insecticide for bednetimpregnation. J Trop Med Hyg 1994;97:325–32.

632. Zeidner M, Shechter M. Reduction of test anxiety: a first attempt at economic evaluation. Anxiety StressCoping 1994;7:1–18.

633. Bate A. Examining the importance of consumerviews and preferences vis-a-vis other criteria that arecommonly used to aid priority setting decisionmaking: a pilot study [MSc Health Economicsthesis]. York: University of York; 1999.

634. Silva MC, Lewis CK. Ethics, policy, and allocation ofscarce resources in nursing service administration: apilot study. Nurs Connections 1991;4(2):44–52.

635. Campbell SM, Hann M, Roland MO, Quayle JA,Shekelle PG. The effect of panel membership andfeedback on ratings in a two-round Delphi survey:results of a randomized controlled trial. Med Care1999;37:964–8.

636. Bradley M. Users manual for SPEED, version 2.1.The Hague: Hague Consulting Group; 1991.

References

96

637. Ham C. Priority setting in the NHS: reports fromsix districts. BMJ 1993;307:435–8.

638. Argyll and Clyde Health Board Priority WorkingGroup. Prioritisation scoring index. Paisley:Department of Public Health, Argyll and ClydeHealth Board; 1998.

639. Aryshire and Arran Health Board Spa Group.Setting Priorities in Ayrshire: report of the Ayrshire and Arran Health Board SPA Group. Ayr: Aryshire and Arran Health Board; 1999.

640. Cumming J. An overview of the health economicsand health policy literatures on priority setting inhealth care. Wellington, New Zealand: HealthServices Research Centre, University of Wellington;1996. Health Services Research Centre ResearchReports no. 4.

641. Dixon J, New B. Setting priorities New Zealand-style[editorial]. BMJ 1997;314:86–7.

642. New B, Le Grand J. Rationing in the NHS:principles and pragmatism. London: King’s Fund;1996.

643. Cumming J. Core services and priority-setting the New Zealand Experience. Health Policy1994;29:41–60.

644. Dumfries and Galloway Health Board. Prioritisation ranking for service developmentproposals. Dumfries: Dumfries and Galloway Health Board; 1998.

645. Feighan T. Setting health care priorities inNorthern Ireland. Work in progress paperpresented at the UK Health Economists’ StudyGroup Meeting; 1998 Jul; Galway.

646. Ham C. Priority setting in health care: learningfrom international experience. Health Policy1997;42:49–66.

647. Honigsbaum F. Who shall live? Who shall die? –Oregon’s health financing proposals. London:King’s Fund College; 1993. King’s Fund CollegePapers No. 4.

648. Hunter D. Desperately seeking solutions: rationingin health care. London: Longman; 1997.

649. Greenhalgh T. How to read a paper: the basics ofevidence based medicine. London: BMJ PublishingGroup; 1997.

650. Stevenson R, Hegarty M. In the picture. Health Serv J 1994;104:22–4.

651. Scottish Association of Health Councils. An analysisof health council effectiveness: representing thepublic in the NHS in Scotland. Edinburgh: ScottishAssociation of Health Councils; 1995.

652. Nitzan S. The vulnerability of point-voting schemesto preference variation and strategic manipulation.Public Choice 1985;47:349–70.

653. LeBreton M, Truchon M. A Borda measure for social choice functions. Math Soc Sci1997;34:249–72.

MEDLINE and HealthSTAR (Ovid)001 exp consumer participation/002 exp consumer satisfaction/003 public opinion/004 ((public or consumer$ or patient$) adj3

(preference$ or opinion or choice$ orparticipat$)).tw

005 or/1–4006 exp data collection/007 exp research008 elicit$.tw009 measure$.tw010 obtain$.tw011 technique$.tw012 or/6–11013 5 and 12014 exp health planning/015 5 and 14016 13 or 15

EMBASE (BIDS – Ovid)

001 ((public or consumer$ or patient$) adj3(preference$ or opinion or choice$ orparticipat$)).tw

002 patient satisfaction/003 patient attitude/004 public opinion/005 or/1–4006 measurement/007 exp information processing/008 elicit$.tw009 obtain$.tw010 measure$.tw011 technique$.tw012 or/6–11013 5 and 12014 consumer/015 healthcare planning/016 resource allocation/017 14 and (15 or 16)018 13 or 17

Social Science Citation Index (BIDS)001 (public or consumer* or patient*) @TKA002 (preference* or choice or opinion or

consult* or participat*) @TKA003 (measur* or elicit* or obtain* or technique*)

@TKA004 1 and 2 and 3

PsycLIT (SilverPlatter)

001 explode “CONSUMER-RESEARCH”002 “PUBLIC-OPINION”003 “PREFERENCES”004 (public or consumer? or patient?) near3

(preference? or choice? or opinion orconsult* or participat*)

005 #1 or #2 or #3 or #4006 explode “METHODOLOGY”007 explode “MEASUREMENT”008 (measure* or elicit* or obtain* or

technique?)009 #6 or #7 or #8010 #5 and #9011 explode “PREFERENCE-MEASURES”012 public or consumer? or patient?013 #11 and #12014 #10 or #13

EconLIT (Ovid)

001 consumer choice.hw002 ((public or consumer$ or patient$) adj3

(preference$ or opinion or choice$ orparticipat$)).tw

003 1 or 2004 (elicit$ or measure$ or method$ or obtain$

or technique$).tw005 3 and 4


97

Appendix 1

Electronic database search strategies for identifying methods for eliciting

public preferences


99

Appendix 2 presents a summary of some of the studies identified in the review to provide

the reader with a summary of the main issuesraised for each of the techniques. This table doesnot provide a listing of all the studies identified in this review.

The only quantitative technique not mentioned in this summary is satisfaction surveys. There are

two reasons for this. First, satisfaction has beenascertained using Likert scales and Guttman scalesand these are covered in the tables. Secondly, ourreview of satisfaction surveys was at a general level,and readers are advised to refer to the body of the text.

Appendix 2

Additional data for chapter 5

Appendix 2

100 Sim

ple

rank

ing

exer

cise

sPr

oper

ties

: th

ere

is n

o w

ell-d

efin

ed t

heo

reti

cal b

asis

for

sim

ple

ran

kin

g ex

erci

ses

and

the

outp

ut o

f su

ch s

tudi

es is

ord

inal

.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

16Bo

wlin

g et

al.,

1993

For

com

mun

ity g

roup

,9 o

f 359

did

not

Con

cern

rai

sed

Giv

en c

ompl

exity

of i

ssue

s,co

mpl

ete

ques

tionn

aire

.For

ran

dom

sam

ple

abou

t hi

gh c

ost

of

pers

onal

inte

rvie

ws

are

the

Publ

ic r

anki

ng o

f 12

heal

thca

re

of p

ublic

,pos

tal q

uest

ionn

aire

initi

ally

gav

e pu

blic

foru

ms

and

only

val

id a

ppro

ach

serv

ices

in C

ity a

nd H

ackn

ey.

resp

onse

rat

e of

11%

.Fol

low

-up

afte

r in

terv

iew

sur

veys

.C

onsi

sted

of c

omm

unity

gro

up

4 re

min

ders

,by

inte

rvie

w,i

ncre

ased

res

pons

e C

ited

to r

ange

di

scus

sion

s,an

d a

rand

om

rate

to

78%

.Res

pons

e ra

tes,

afte

r 4

mai

lings

,be

twee

n £2

0,00

0 sa

mpl

e of

the

pub

lic a

nd

was

66%

(13

1/19

7) fo

r th

e do

ctor

s,68

%

and

£90,

000

doct

ors.

Mor

e de

tail

of

(82/

121)

for

the

cons

ulta

nts

and

6/7

of

thes

e su

rvey

s in

Bow

ling36

6th

e pu

blic

hea

lth d

octo

rs

358

Shic

kle,

1997

Whi

lst

resp

onde

nts

A pr

iori

hypo

thes

es c

onfir

med

:ga

ve h

igh

rank

ings

to

pref

eren

ces

for

life-

savi

ng t

reat

men

ts,

Rev

iew

of p

ublic

pre

fere

nces

lif

e-sa

ving

tre

atm

ent,

trea

ting

the

youn

g,pa

tient

s w

ithfo

r pr

iori

tisat

ion

in t

he U

Kin

a b

inar

y ch

oice

de

pend

ants

,and

tho

se w

ho h

adth

ey s

tate

d a

pref

-le

d a

heal

thy

lifes

tyle

eren

ce fo

r qu

ality

-of

-life

impr

ovin

gtr

eatm

ents

18Fu

rnha

m e

t al

.,19

98R

espo

nse

rate

by

ques

tionn

aire

tak

en h

ome

A pr

iori

hypo

thes

es w

ere

conf

irm

edw

ith p

atie

nts

was

81%

– fe

mal

es w

ere

favo

ured

ove

r m

ales

,Pu

blic

pri

oriti

satio

n of

kid

ney

non-

smok

ers

over

sm

oker

s,an

ddi

alys

is p

atie

nts

diffe

ring

by

sex,

poor

peo

ple

over

ric

h.Fi

ndin

gsag

e,in

com

e,dr

inki

ng h

abits

ag

reed

with

Fur

nham

and

Bri

ggs26

and

relig

ious

bel

iefs

24A

nger

mey

er e

t al

.,19

99R

epre

sent

ativ

e su

rvey

,res

pons

e ra

te w

as

Ran

king

led

to m

ore

prom

inen

t71

.2%

(15

64 in

terv

iew

s)di

ffere

nces

bet

wee

n th

e po

tent

ial

Publ

ic a

ttitu

des

tow

ards

so

urce

s of

hel

p th

an r

atin

g sc

ales

trea

tmen

t of

dep

ress

ion.

From

8

trea

tmen

ts,r

espo

nden

ts

Stra

nge

that

men

tal h

ealth

wer

e as

ked

to p

rior

itise

top

2pr

ofes

sion

als

do n

ot p

lay

a ro

le in

help

-see

king

in m

ajor

dep

ress

ion

Ran

king

is a

clo

ser

refle

ctio

n of

re

ality

sin

ce p

eopl

e ar

e fo

rced

to

mak

e de

cisi

ons

as t

hey

do in

dai

ly li

fe

cont

inue

d


101Sim

ple

rank

ing

exer

cise

s co

ntd

Prop

erti

es: t

her

e is

no

wel

l-def

ined

th

eore

tica

l bas

is f

or s

impl

e ra

nki

ng

exer

cise

s an

d th

e ou

tput

of

such

stu

dies

is o

rdin

al.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

25Fu

rnha

m,1

996

95%

(15

9/16

7) r

espo

nse

rate

A pr

iori

hypo

thes

es c

onfir

med

(an

d in

line

with

oth

er s

tudi

es):

resp

onde

nts

Peop

le’s

rank

ord

erin

g (in

fa

vour

ed fe

mal

es o

ver

mal

es,p

oor

term

s of

4 d

imen

sion

s:ge

nder

,ov

er r

ich,

and

non-

drin

kers

ove

rin

com

e,al

coho

l con

sum

ptio

n dr

inke

rs.18

,26T

hey

also

favo

ured

and

relig

ious

bel

iefs

) of

tre

atm

ent

Chr

istia

ns o

ver

athe

ists

for

patie

nts

suffe

ring

from

ki

dney

failu

re

366

Bow

ling,

1999

Pilo

t in

dica

ted

resp

onde

nts

foun

d it

diffi

cult

Stud

y di

d no

t ob

tain

The

con

sist

ency

N

ot p

ossi

ble

toH

ighe

r pr

iori

ty fo

r lif

e-sa

ving

to r

ank

all 1

2 op

tions

– t

here

fore

cat

egor

ised

the

coop

erat

ion

ofof

the

pub

lic’s

com

pare

pilo

t an

dtr

eatm

ents

and

low

er p

rior

ity fo

rM

ore

deta

iled

repo

rt o

f City

in

to a

4-c

ateg

ory

rank

ing

met

hod

the

loca

l hea

lthpr

iori

tisat

ion

ofm

ain

stud

ies

due

heal

th e

duca

tion

and

fam

ily p

lann

ing

and

Hac

kney

pub

lic p

rior

ity

(‘ess

entia

l’,‘v

ery

impo

rtan

t’,‘im

port

ant’,

coun

cil w

hom

enta

l hea

lthto

rew

ordi

ng in

co

nsis

tent

with

ear

lier

wor

k73

rank

ing

of 1

2 he

alth

care

‘le

ss im

port

ant’)

prot

este

d ab

out

serv

ices

as

ath

e qu

estio

nnai

re.

serv

ices

.16C

omm

unity

gro

up

the

ethi

cs o

f it

med

ium

pri

ority

Nev

erth

eles

s,te

st–

For

com

mun

ity g

roup

,pri

oriti

satio

n di

scus

sion

s,a

rand

om s

ampl

e M

ain

stud

y co

mpr

ised

350

res

pond

ents

.w

as c

onfir

med

in a

rete

st r

elia

bilit

y of

men

tal h

ealth

and

pre

vent

ativ

eof

the

pub

lic a

nd d

octo

rsW

hils

t re

spon

se r

ate

to t

he p

osta

l sur

vey

The

stu

dyco

nsis

tenc

y ch

eck

was

che

cked

at

serv

ices

con

sist

ent

with

5-it

emw

as p

oor,1

6 th

e fo

llow

-up

by le

tter

and

(c

omm

unity

gro

ups,

agai

nst

addi

tiona

lth

e pi

lot

stag

e an

d pr

iori

ty li

st (

sign

ifica

nt a

t 5%

leve

l)in

terv

iew

was

goo

d.R

espo

nse

rate

s fo

r th

e po

stal

and

inte

rvie

wat

titud

e qu

estio

nsal

mos

t id

entic

al

doct

ors

wer

e 66

% fo

r th

e co

nsul

tant

s,68

%

surv

eys)

too

kra

nkin

gs w

ere

For

rand

om s

ampl

e of

pub

lic,m

enta

lfo

r th

e G

Ps a

nd 6

/7 o

f the

pub

lic h

ealth

12

mon

ths

toD

octo

rs w

ere

aske

dfo

und

heal

th p

rior

itisa

tion,

prev

enta

tive

doct

ors

afte

r 4

mai

lings

com

plet

e an

d an

OE

ques

tion

tose

rvic

es,a

nd a

nsw

ers

to t

he 5

-item

cost

£24

,000

list

in o

rder

of

prio

ritie

s lis

t si

gnifi

cant

ly c

orre

late

d15

res

pond

ents

gav

e ne

gativ

e or

cri

tical

pr

iori

ty 5

are

as o

fat

5%

leve

lco

mm

ents

abo

ut t

he e

xerc

ise.

Res

pond

ents

im

prov

emen

t th

eyfo

und

the

exer

cise

ver

y di

fficu

lt an

d th

e w

ould

like

in C

ityR

espo

nses

sen

sitiv

e to

que

stio

nw

ordi

ng m

ay r

equi

re fu

rthe

r si

mpl

ifica

tion

and

Hac

kney

.The

irw

ordi

ng,e

.g.i

n pi

lot “

inte

nsiv

e ca

rere

spon

ses

wer

efo

r pr

emat

ure

babi

es”

was

giv

enR

esea

rche

rs m

ay h

ave

to g

o ou

t an

d in

terv

iew

co

nsis

tent

with

high

est

prio

rity

1 bu

t w

hen

qual

ified

peop

le in

dep

rive

d ar

ea r

athe

r th

an r

elyi

ng

thei

r ra

nkin

gsw

ith s

tate

men

t “w

eigh

ing

less

tha

non

the

m t

o re

turn

que

stio

nnai

res

11/2

pou

nds

and

unlik

ely

to s

urvi

ve”

rank

dro

pped

to

10Is

sue

too

com

plex

for

post

al q

uest

ionn

aire

s

Appendix 2

102 Qua

litat

ive

disc

rim

inan

t pr

oce

ss (

QD

P)

Prop

erti

es: Q

DP

has

its

theo

reti

cal b

asis

in d

ecis

ion

th

eory

, fuz

zy s

et t

heo

ry, t

he

theo

ry o

f va

gue

real

num

bers

an

d vo

tin

g th

eory

. Giv

en t

he

map

pin

g of

qual

itat

ive

resp

onse

s on

to a

n in

terv

al s

cale

, th

e te

chn

ique

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

28Br

yson

et

al.,

1994

Aut

hors

not

e th

e te

chni

que

is “

sim

ple

and

Com

pute

r-ba

sed

soin

tuiti

vely

app

ealin

g”.T

hey

also

not

e its

eas

ere

lativ

ely

expe

nsiv

eD

evel

opm

ent

of Q

DP

met

hod

of u

se in

sof

twar

e fo

rmto

col

lect

dat

aw

ith il

lust

ratio

n of

iden

tifyi

ng

suita

ble

cand

idat

es fo

r th

e po

sitio

n of

dea

n of

the

sc

hool

of b

usin

ess

30N

gwen

yam

a &

Bry

son,

1998

One

of t

he 3

exp

erts

gave

inco

nsis

tent

Des

crip

tion

of Q

DP

met

hod

answ

ers

whi

ch,o

nce

with

illu

stra

tive

exam

ple

on

brou

ght

to h

is/h

erth

e di

agno

sis,

by 3

exp

erts

,at

tent

ion,

wer

eof

a p

atie

nt w

hose

sym

ptom

s co

rrec

ted

by t

hein

dica

te 4

pos

sibl

e di

seas

es:

expe

rt.C

onsi

sten

tga

stri

c ca

ncer

,pan

crea

tic

answ

ers

are

in t

his

canc

er,f

unct

iona

l dis

orde

r,w

ay a

ssur

ed w

ithan

d ga

llsto

nes

this

met

hod


103CA

ran

king

exe

rcis

esPr

oper

ties

: th

e C

A r

anki

ng

appr

oach

was

dev

elop

ed in

mat

hem

atic

al p

sych

olog

y fr

om c

onjo

int

mea

sure

men

t th

eory

.32It

is a

lso

root

ed in

Lan

cast

er’s

th

eory

of v

alue

.31T

he

theo

reti

cal u

nde

rpin

nin

g re

sult

s in

a s

tren

gth

of

pref

eren

ce m

easu

re.39

8

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

37C

hinb

urap

a et

al.,

1993

51/6

3 ph

ysic

ians

agr

eed

to p

artic

ipat

eTo

incr

ease

val

idity

res

pond

ents

(o

ffere

d $5

0 to

do

so).

3 w

ere

excl

uded

:in

the

exp

erim

enta

l gro

up w

ere

told

D

eter

min

ing

phys

icia

n’s

2 m

ade

erro

rs a

nd 1

ref

used

to

com

plet

eth

eir

deci

sion

s w

ould

be

revi

ewed

de

cisi

on-m

akin

g pr

oces

s in

th

e ta

sk.A

vera

ge c

ompl

etio

n tim

e w

asby

pee

rs;h

owev

er,t

his

did

not

cont

ext

of a

nti-i

nfec

tive

drug

s35

min

utes

to

rank

18

drug

pro

files

influ

ence

res

pons

es

Phys

icia

ns s

hift

ed fr

om u

sing

co

mpe

nsat

ory

to n

on-c

ompe

nsat

ory

deci

sion

-mak

ing

as c

ompl

exity

of

task

incr

ease

d

41O

rkin

& G

reen

how

,197

834

facu

lty m

embe

rs;r

espo

nden

ts s

tate

d ta

skR

anki

ngs

corr

elat

ed h

ighl

y w

ithw

as d

iffic

ult

beca

use

it fo

rced

the

m t

o th

ink

rank

ings

pre

dict

ed o

n th

e ba

sis

ofFa

culty

mem

bers

’ eva

luat

ion

abou

t re

lativ

e im

port

ance

of c

rite

ria

thei

r se

ts o

f util

ities

(m

ean

0.93

)of

res

iden

ts’ c

linic

al c

ompe

tenc

e in

ter

ms

of 6

att

ribu

tes

42Pa

rker

& S

rini

vasa

n,19

76D

oor-

to-d

oor

inte

rvie

ws

yiel

ded

a re

spon

seR

esul

ts c

ompa

red

rate

of 9

0.3%

(17

7/19

6).2

5 fa

cilit

y pr

ofile

sw

ith a

sec

ond

set

Pref

eren

ces

for

rura

l w

ere

divi

ded

into

“lik

e” a

nd “

don’

t lik

e” a

ndof

pro

files

wer

e he

alth

care

sys

tem

then

ran

ked.

Thi

s pr

oces

s to

ok 1

2–18

min

utes

not

stat

istic

ally

si

gnifi

cant

at

0.10

le

vel (

n=

8 a

t 2

mon

ths

afte

r in

itial

sur

vey)

43R

osko

et

al.,

1983

Con

veni

ence

sam

ple

of 7

3 st

uden

ts.R

anke

dIn

a c

ompa

rativ

eC

oeffi

cien

ts fo

r th

e le

vels

of a

ttri

bute

s18

pro

files

.The

y co

nduc

ted

a se

cond

ran

king

test

,usi

ng 7

(in t

erm

s of

cha

rges

,tra

vel t

ime,

offic

ePr

efer

ence

s fo

r ex

erci

se w

here

2 a

ttri

bute

s co

uld

not

sepa

rate

pro

files

,op

enin

g ho

urs,

wai

ting

time,

park

ing

ambu

lato

ry c

are

co-e

xist

.Use

d or

thog

onal

arr

ay t

o lim

it pr

edic

ted

rank

sfa

cilit

ies,

type

of p

ract

ice

and

spon

sor)

man

agem

ent

the

scen

ario

s to

27

wer

e w

ithin

1 r

ank

all b

ehav

ed in

line

with

a p

riori

of t

he a

ctua

l ex

pect

atio

ns.A

utho

rs n

ote

that

the

obse

rved

in t

he

valid

ity a

nd r

elia

bilit

y of

CA

hav

e be

enm

ain

stud

yw

ell e

stab

lishe

d in

oth

er d

isci

plin

es

but

need

ass

essi

ng in

hea

lthca

re cont

inue

d

Appendix 2

104 CA

ran

king

exe

rcis

es c

ontd

Prop

erti

es: t

he

CA

ran

kin

g ap

proa

ch w

as d

evel

oped

in m

ath

emat

ical

psy

chol

ogy

from

con

join

t m

easu

rem

ent

theo

ry.32

It is

als

o ro

oted

in L

anca

ster

’s t

heo

ryof

val

ue.31

Th

e th

eore

tica

l un

derp

inn

ing

resu

lts

in a

str

engt

h o

f pr

efer

ence

mea

sure

.398

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

21R

osko

& M

cKen

na,1

983

Con

veni

ence

sam

ple

Mor

e th

an 8

0% o

fC

oeffi

cien

ts fo

r th

e le

vels

of

rank

s w

ere

with

inat

trib

utes

(in

ter

ms

of c

harg

es,t

rave

l Pr

efer

ence

s fo

r al

tern

ativ

e 1

rank

of t

he a

ctua

ltim

e,of

fice

open

ing

hour

s,w

aitin

g he

alth

care

pla

ns (

26 p

rofil

e pr

ofile

whe

n a

time,

park

ing

faci

litie

s,ty

pe o

f ca

rds)

.Als

o,tr

ade-

off a

ppro

ach,

seco

nd r

anki

ng o

fpr

actic

e an

d sp

onso

r) a

ll be

have

d w

hich

con

sist

s of

pai

red

7 pr

ofile

car

ds w

asin

line

with

a p

riori

expe

ctat

ions

com

pari

sons

of d

iffer

ent

atte

mpt

ed.T

hele

vels

of a

ttri

bute

str

ade-

off a

ppro

ach

yiel

ded

low

er

repr

oduc

ibili

ty o

r co

mpa

rativ

e va

lidity

44Sh

emw

ell &

Yav

as,1

997

The

coe

ffici

ents

for

the

attr

ibut

es(c

ost,

qual

ity o

f nur

sing

car

e,qu

antit

yPa

tient

s’ p

refe

renc

es fo

r a

of r

ecre

atio

nal f

acili

ties,

qual

ity o

fco

ngre

gate

car

e he

alth

faci

lity

phys

ical

faci

litie

s,lo

catio

n,qu

ality

of

reha

bilit

atio

n pr

ogra

mm

es a

nd s

taff

attit

udes

) be

have

d as

exp

ecte

d

46Si

ngh

et a

l.,19

98C

onse

nt r

ate

for

inte

rvie

ws

was

83.

7%5

resp

onde

nts

Rel

iabi

lity

The

coe

ffici

ents

for

the

attr

ibut

es(1

59/1

90).

Pret

est

inte

rvie

wee

s ha

d lit

tlega

ve 3

or

mor

eco

effic

ient

s of

0.7

(mag

nitu

de o

f effe

ct,c

erta

inty

of

Pref

eren

ces

for

grow

th

diffi

culty

with

the

CA

tas

kin

cons

iste

nt a

nsw

ers,

and

0.66

wer

e fo

und

effe

ct,t

reat

men

t ro

ute,

cost

s,si

de-

augm

enta

tion

ther

apy

usin

g C

A11

.95%

gav

e 2

and

for

10 r

espo

nden

tsef

fect

s,ch

ild’s

attit

ude)

beh

aved

as

40.2

5% g

ave

1(a

fter

7-m

onth

gap

)ex

pect

ed.O

ther

a p

riori

hypo

thes

es

wer

e su

ppor

ted:

the

risk

-con

scio

us

grou

p w

ere

mos

t in

fluen

ced

by s

ide-

effe

cts;

the

child

-focu

sed

grou

p w

ere

mos

t in

fluen

ced

by t

he c

hild

’s at

titud

e;th

e co

st-c

onsc

ious

gro

up

wer

e m

ost

influ

ence

d by

out

-of-

pock

et c

osts

;and

the

eas

e-of

-use

gr

oup

wer

e m

ost

influ

ence

d by

rou

te

of t

reat

men

t.In

add

ition

,the

cos

t-co

nsci

ous

grou

p w

ere

char

acte

rise

d by

low

er in

com

e an

d ed

ucat

ion,

and

the

child

-focu

sed

pare

nts

wer

e ch

arac

teri

sed

by m

oder

ate

leve

ls

of in

com

e an

d ed

ucat

ion


105Vis

ual a

nalo

gue

scal

e (V

AS

)Pr

oper

ties

: th

e VA

S h

as it

s th

eore

tica

l bas

e in

psy

chom

etri

cs.24

8,39

9–40

2T

her

e is

deb

ate

con

cern

ing

wh

eth

er t

he

VAS

mea

sure

s st

ren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

253

Braz

ier

et a

l.,19

99VA

S m

etho

d w

idel

y se

en a

s th

e m

ost

feas

ible

Gud

ex e

t al

.359

cite

dT

he fo

llow

ing

Supp

orte

rs o

f VA

S ar

gue

the

outp

utan

d ac

cept

ed o

f the

hea

lth-s

tate

val

uatio

n57

.4%

of r

espo

nden

tspr

esen

ts t

est–

rest

is c

ardi

nal.14

,59,

407,

412

How

ever

,oth

ers

Res

ults

of a

sys

tem

atic

rev

iew

te

chni

ques

had

no lo

gica

lre

liabi

lity

resu

lts fo

rch

alle

nge

this

.61,6

2,24

6,39

9,61

7

of t

he u

se o

f hea

lth s

tatu

s in

cons

iste

ncie

sth

eVA

S:m

easu

res

in e

cono

mic

Hig

h re

spon

se r

ates

and

hig

h le

vels

of

Valid

ity o

fVA

S ch

alle

nged

on

basi

sev

alua

tion

com

plet

ion35

2,35

9,40

0,40

6–41

1M

ean

over

all

1 w

eek

or le

ss =

that

pre

fere

nces

are

elic

ited

unde

rin

cons

iste

ncy

rate

0.77

* ;0.7

0–0.

95†

cert

aint

y.H

owev

er,D

yer

and

Gud

ex e

t al

.359

repo

rts

that

of 3

395

resp

on-

was

2.5

%.M

ean

4 w

eeks

=Sa

rin42

6su

gges

t a

mea

sura

ble

valu

ede

nts

only

3.2

% o

f res

pons

es w

ere

excl

uded

logi

cal i

ncon

sist

ency

0.62

‡ ;0.8

9§fu

nctio

n pr

ovid

ing

a lin

k be

twee

nfr

om a

naly

sis,

whi

lst

Silv

erts

sen

et a

l.410

and

at r

etes

t w

as 2

.2%

,10

wee

ks =

0.7

8#su

ch v

alue

and

util

ityFe

rraz

et

al.61

6re

port

com

plet

ion

rate

s of

sl

ight

ly le

ss t

han

1 ye

ar =

0.4

9¶

95 a

nd 1

00%

,res

pect

ivel

yat

tes

tT

heVA

S m

etho

d m

ay b

e su

scep

tible

Cor

rela

tions

und

er-

to r

espo

nse-

spre

adin

g,51

,407

,425

furt

her

The

VAS

met

hod

is c

heap

er a

nd q

uick

er

Dol

an a

nd K

ind36

0ta

ken

whe

re s

peci

fied:

chal

leng

ing

the

inte

rval

pro

pert

ies

ofth

an o

ther

met

hods

(gi

ven

that

it c

an b

e fo

und

inco

nsis

tenc

yin

trac

lass

cor

rela

tion

the

tech

niqu

e.K

apla

n et

al.40

7

carr

ied

out

in a

mai

led

ques

tionn

aire

),bu

t ra

tes

of a

roun

d 10

%co

effic

ient

(IC

C)†,

‡,#

;re

cogn

ise

the

pote

ntia

l for

res

pons

e-is

not

nec

essa

rily

eas

ier

to c

ompl

ete40

0,41

4fo

r fir

st s

tudy

,whi

lst

Pear

son

corr

elat

ion

spre

adin

g bi

ases

.How

ever

,the

y st

ate

ever

y su

bsam

ple

coef

ficie

nt§ ;o

ther

sth

at s

uch

bias

es m

ay b

e co

ntro

lled

of t

he s

econ

d un

spec

ified

for

by v

alui

ng 1

sta

te a

t a

time

orpr

oduc

ed m

edia

n th

roug

h th

e us

e of

a b

alan

ced

desi

gnra

tes

belo

w 3

%G

udex

et

al.35

9m

ean

ICC

= 0

.78.

Onl

y VA

S m

etho

ds w

ere

gene

rally

foun

d13

res

pond

ents

had

to

hav

e a

wea

k co

rrel

atio

n w

ith

an IC

C <

0.6

SG a

nd T

TO.41

4,42

2,46

9,61

8C

urre

ntly,

insu

ffici

ent

evid

ence

to

infe

r an

y co

rrel

atio

n be

twee

n th

e VA

S or

PTO

.VA

S m

etho

ds c

orre

late

wel

l with

m

easu

res

of h

ealth

sta

tus

(e.g

.pai

n,cl

inic

al s

ympt

oms,

etc.

) co

mpa

red

with

SG

and

TTO

* O

’Con

nor41

8 ;† Ba

kker

et

al.40

9 ;‡ O

’Brie

n an

d Vi

ram

onte

s419 ;§

Gab

riel e

t al

.420 ;#

Gud

ex e

t al

.359 ;¶

Torr

ance

400

Appendix 2

106 Rat

ing

scal

es w

ithi

n C

APr

oper

ties

: th

e C

A r

atin

g ap

proa

ch h

as it

s th

eore

tica

l bas

is in

info

rmat

ion

inte

grat

ion

th

eory

an

d ju

dgem

ent

anal

ysis

,427,

428

wh

ere

agai

n it

has

bee

n a

rgue

dth

at c

ardi

nal

dat

a ca

n b

e ob

tain

ed f

rom

indi

vidu

al r

espo

nse

s to

rat

ing

data

. In

add

itio

n, C

A r

atin

g ex

erci

ses

are

root

ed in

Lan

cast

er’s

th

eory

of

valu

e.31

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

52C

hakr

abor

ty e

t al

.,19

93In

cent

ive

of $

5 fo

r co

mpl

etio

n.3

prac

tice

Aut

hors

not

e ch

oice

s ar

e ea

sier

for

prof

iles

wer

e us

ed t

o ge

t re

spon

dent

s us

edre

spon

dent

s an

d re

sem

ble

real

-life

Patie

nts’

pre

fere

nces

for

to t

he fo

rmat

.42%

res

pons

e ra

te (

126/

300)

situ

atio

nsde

ntal

ser

vice

s us

ing

CA

w

ith p

osta

l que

stio

nnai

re a

nd 1

follo

w-u

p(1

0-po

int

scal

e)

53C

hinb

urap

a &

Lar

son,

1988

44%

res

pons

e ra

te (

43/9

8) w

ith p

osta

lqu

estio

nnai

re a

fter

2 r

emin

ders

;12

prof

iles

The

impo

rtan

ce o

f dru

g at

trib

utes

in p

hysi

cian

’s pr

escr

ibin

g

54D

iam

ond

et a

l.,19

9450

% r

espo

nse

rate

(10

4/20

9) fr

om 2

mai

lings

In t

erm

s of

att

ribu

tes,

the

auth

ors

and

tele

phon

e fo

llow

-up

note

“if

addi

tiona

l fac

tors

wer

eSp

ecia

lty s

elec

tion

by fo

urth

ad

ded,

or s

ome

...w

ere

dele

ted,

year

med

ical

stu

dent

s us

ing

CA

a di

ffere

nt p

atte

rn o

f fin

ding

s m

ight

em

erge

”

56G

raf e

t al

.,19

93A

utho

rs n

ote

CA

can

res

ult

in fa

tigue

.Rat

ing

16–2

4 pr

ofile

s to

ok a

roun

d 10

–20

min

utes

Use

of C

A in

des

ign

of a

n ob

stet

rics

uni

t (1

0-po

int

scal

e)Te

leph

one

inte

rvie

ws

diffi

cult

and

the

tech

niqu

e be

st s

uite

d to

face

-to-

face

in

terv

iew

s

Mai

l sur

veys

pos

sibl

e w

here

ince

ntiv

es

are

offe

red

58H

arw

ood

et a

l.,19

94R

espo

nse

rate

42%

(10

1/24

0) a

nd 7

9% o

fR

elia

bilit

y te

sted

in a

thes

e (7

9) c

ompl

eted

the

inte

rvie

wpi

lot

of 9

res

pond

ents

Dev

elop

ing

an o

utco

me

with

a 2

-wee

k re

test

.m

easu

re fo

r ha

ndic

apA

utho

rs n

oted

tha

t in

terv

iew

s “w

ere

prob

ably

D

iffer

ence

s of

with

inab

out

as d

iffic

ult

as it

is r

easo

nabl

e to

1

cate

gory

foun

d.un

dert

ake”

,tho

ugh

the

resp

onse

rat

e w

as

5 ‘te

st’ s

cena

rios

not

com

para

ble

with

oth

er s

calin

g m

etho

dsus

ed in

the

mai

n st

udy

dem

onst

rate

d go

od

agre

emen

t w

ith t

he

mai

n su

rvey

:Pea

rson

0.

98 a

nd K

enda

ll 1.

00

cont

inue

d


107Rat

ing

scal

es w

ithi

n C

A c

ontd

Prop

erti

es: t

he

CA

rat

ing

appr

oach

has

its

theo

reti

cal b

asis

in in

form

atio

n in

tegr

atio

n t

heo

ry a

nd

judg

emen

t an

alys

is,42

7,42

8w

her

e ag

ain

it h

as b

een

arg

ued

that

car

din

al d

ata

can

be

obta

ined

fro

m in

divi

dual

res

pon

ses

to r

atin

g da

ta. I

n a

ddit

ion

, CA

rat

ing

exer

cise

s ar

e ro

oted

in L

anca

ster

’s t

heo

ry o

f va

lue.

31

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

59R

eard

on &

Pat

hak,

1990

143/

168

resp

onde

nts

prov

ided

Not

ed t

hat

indi

vidu

als

will

not

be

usea

ble

resp

onse

sab

le t

o de

al w

ith a

larg

e nu

mbe

r of

Pref

eren

ces

for

prod

uct

attr

ibut

es.4

7 at

trib

utes

iden

tifie

d at

trib

utes

for

antih

ista

min

e an

d re

duce

d to

7 b

y a

pane

l of

drug

s10

pat

ient

s

61R

yan

et a

l.,19

9867

% r

espo

nse

rate

(67

/100

);Ev

iden

ce o

f int

erna

lEv

iden

ce o

f int

erna

l val

idity

(e.

g.50

ans

wer

ed t

he C

A q

uest

ion

cons

iste

ncy

ofw

aitin

g ha

d a

nega

tive

sign

,and

Pref

eren

ces

for

patie

nt h

ealth

be

twee

n 80

% a

ndbe

tter

off

peop

le h

ad h

ighe

rca

rd (

scal

e of

1–8

)96

% (

with

‘bet

ter’

op

port

unity

cos

t of

tim

e).N

osc

enar

ios

bein

g ev

iden

ce o

f ord

erin

g ef

fect

s in

ra

ted

mor

e hi

ghly

)te

rms

of s

cena

rio

orde

ring

With

in t

rial

,onl

y th

ose

resp

onde

nts

who

had

exp

erie

nce

of a

pat

ient

he

alth

car

d va

lued

it

64W

igto

n et

al.,

1986

Aut

hors

not

e th

at v

igne

ttes

are

con

veni

ent

3 in

divi

dual

s te

sted

Pred

icte

d an

swer

s fo

r es

timat

edan

d ea

sy t

o ad

min

iste

rfo

r re

prod

ucib

ility

–m

odel

wer

e co

rrec

t 90

% o

f the

tim

ePh

ysic

ians

’ wei

ghtin

gs o

f typ

es

foun

d cl

ose

clus

teri

ngs

of c

linic

al in

form

atio

n us

ed in

of

bet

a w

eigh

tsdi

agno

sing

pul

mon

ary

embo

lism

Appendix 2

108 Sch

edul

e fo

r th

e ev

alua

tio

n o

f in

divi

dual

qua

lity

of

life

(SE

IQo

L)

Prop

erti

es: s

har

es t

he

sam

e th

eore

tica

l bas

is a

s ra

tin

g sc

ale

CA

(se

e pa

ges

106–

107)

.

Ref

.S

tudy

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

alR

epro

duci

bilit

yV

alid

ity†

cons

iste

ncy*

70M

cGee

et

al.,

1998

Cor

rect

com

plet

ion

by 1

00%

,but

by

inte

rvie

wr

= 0

.74

for

heal

thy

R2

= 0

.75

for

heal

thy

resp

onde

nts

not

self-

com

plet

ion

resp

onde

nts

SEIQ

oL in

hea

lthy

and

R2

= 0

.79

for

outp

atie

nts

gast

roen

tero

logy

pop

ulat

ions

r=

0.5

4 fo

r ou

tpat

ient

s

68O

’Boy

le e

t al

.,19

9210

/30

patie

nts

decl

ined

to

take

par

t or

Test

–ret

est

Diff

eren

ces

betw

een

patie

nts

and

coul

d no

t co

mpl

ete

coef

ficie

nt =

–0.

88co

ntro

ls;S

EIQ

oL b

ehav

ed s

imila

rly

Mea

suri

ng q

ualit

y of

life

in

to s

tand

ard

scal

es (

e.g.

AIM

)hi

p re

plac

emen

t pa

tient

s us

ing

SEIQ

oL

67C

oen

et a

l.,19

93O

nly

6 of

20

com

plet

edr

= 0

.74

R2

= 0

.70

‡ r

= 0

.75,

r=

0.6

6‡ R

2=

0.7

,R2

=0.

72U

se o

f SEI

QoL

to

mea

sure

qu

ality

of l

ife in

mild

dem

entia

pa

tient

s

69Br

owne

et

al.,

1994

10%

non

-com

plet

ion

(from

67

resp

onde

nts)

r=

0.7

4 fo

r el

icite

dR

2=

0.7

5 fo

r el

icite

d an

d 0.

79in

terv

iew

ed a

t ho

me.

Infe

rs 9

0% a

ble

and

0.69

for

prov

ided

for

prov

ided

Use

of S

EIQ

oL t

o m

easu

re

to u

nder

stan

dqu

ality

of l

ife in

the

hea

lthy

elde

rly

* In

tern

al c

onsis

tenc

y m

easu

red

usin

g ju

dgem

ent

relia

bilit

y co

effic

ient

† In

tern

al (

cons

truc

t) va

lidity

has

bee

n as

sess

ed b

y as

sess

ing

whe

ther

the

var

ianc

e in

qua

lity

of li

fe ju

dgem

ents

is e

xpla

ined

by

the

set

of c

ues.

This

is m

easu

red

by R

2(w

hich

is a

mea

sure

of t

he c

orre

latio

n be

twee

ntw

o va

riabl

es)

‡ U

npub

lishe

d re

sults

rep

orte

d in

Coe

n et

al.67


109Lik

ert

scal

esPr

oper

ties

: th

ere

is n

o co

nst

rain

ed c

hoi

ce in

Lik

ert

scal

es, m

ean

ing

that

res

pon

den

ts m

ay b

e en

cour

aged

to

over

stat

e pr

efer

ence

s. D

ebat

e ex

ists

on

wh

eth

eran

ord

inal

or

card

inal

mea

sure

is e

stim

ated

.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

80M

arks

et

al.,

1992

283

ques

tionn

aire

s co

mpl

eted

,red

uced

to

Cro

nbac

h’s

α=

0.9

2,G

ood

test

–ret

est

Wea

k co

rrel

atio

n in

exp

ecte

d19

7 w

hen

excl

udin

g m

issi

ng d

ata

n=

77,

outp

atie

nts

relia

bilit

y,in

trac

lass

di

rect

ion

for

3 m

edic

al in

dica

tors

of

Qua

lity

of li

fe in

pat

ient

s co

rrel

atio

n =

0.8

0 as

thm

a se

veri

ty,s

uppo

rtin

g va

lidity

with

ast

hma

20-it

em q

uest

ionn

aire

com

plet

ed in

less

C

ronb

ach’

s α

= 0

.94,

for

58 r

espo

nden

tsth

an 5

min

utes

n=

87,

com

mun

ity

Whi

lst

the

7-po

int

Like

rt s

cale

has

sam

ple

been

sho

wn

to b

e co

mpa

rabl

e to

5-po

int

Like

rt s

cale

use

d be

caus

e “e

asie

r VA

S,th

e 5-

poin

t Li

kert

,how

ever

,to

adm

inis

ter

and

inte

rpre

t” t

han

a VA

Sha

s no

t be

en d

irec

tly c

ompa

red

83Pf

enni

ngs

et a

l.,19

9517

7 st

uden

ts r

espo

nded

,9 in

clud

ed it

emLi

kert

sco

res

wer

e tr

ansf

orm

ed in

tono

n-re

spon

se a

nd w

ere

excl

uded

from

100-

poin

t VA

S.In

3 o

f 9 it

ems,

mea

nA

com

pari

son

of L

iker

t an

d th

e an

alys

isw

as s

igni

fican

tly h

ighe

r th

an V

AS,

VAS

thro

ugh

med

ical

stu

dent

s’

for

the

othe

r 6

(3 o

f whi

ch w

ere

opin

ion

of p

reco

nditi

ons

sign

ifica

nt) V

AS

scor

es w

ere

grea

ter.

of r

espo

nsiv

enes

sA

utho

rs c

oncl

ude

that

the

ir r

esul

ts

prov

ide

mod

est

supp

ort

that

VA

S is

pr

efer

able

to

Like

rt s

cale

s

77Ty

mst

ra &

And

ela,

1993

Res

pons

e ra

te w

as 9

5.8%

.Pilo

t re

spon

se

rate

s w

ere

72.8

% (

of 4

71)

nurs

es a

nd

Doc

tors

’ and

nur

ses’

opi

nion

s 52

.7%

of d

octo

rsof

hea

lthca

re p

olic

y,ra

tioni

ng

and

tech

nolo

gy in

The

N

ethe

rlan

ds.5

-poi

nt r

atin

g sc

ale

from

‘ver

y ad

equa

te’

to ‘v

ery

poor

’

Appendix 2

110 Sem

anti

c di

ffer

enti

al t

echn

ique

(S

DT

)Pr

oper

ties

: th

e te

chn

ique

off

ers

no

con

stra

ined

ch

oice

, no

stre

ngt

h o

f pr

efer

ence

or

any

kin

d of

th

eore

tica

l bas

is.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

93G

irón

& G

omez

bene

yto,

1995

90 r

espo

nse

rate

with

long

-ter

m

Aut

hors

sug

gest

“th

e se

man

ticfo

llow

-up

(2 y

ears

)di

ffere

ntia

l doe

s no

t m

easu

re t

heSt

udy

of fa

mily

att

itude

s an

d re

lativ

e’s

attit

ude

but

thei

r ac

cura

cyre

laps

e in

sch

izop

hren

iain

ass

essi

ng ..

.”,i.e

.thi

s pa

rtic

ular

te

chni

que

does

not

mea

sure

wha

t it

shou

ld m

easu

re

89Bo

wle

s,19

86H

ighl

y ac

cept

able

Cro

nbac

h’s

α=

0.9

6

Att

itude

s to

war

ds t

he

men

opau

se

85H

olm

es,1

974

Test

–ret

est

sugg

ests

Q

uest

ionn

aire

-effe

ct a

ttitu

des

are

that

“th

e ty

pica

l rat

ing

mod

ified

som

ewha

t fr

om in

itial

mor

eSt

atis

tical

eva

luat

ion

of r

atin

g sc

ales

are

not

as

extr

eme

resp

onse

s an

d w

ithin

pag

esc

ales

in m

arke

ting

relia

ble

as w

e w

ould

bi

as:r

espo

nden

ts p

refe

r th

e le

ft-h

and

like

them

to

be”

side

of p

age

429

App

els

et a

l.,19

96D

ata

on la

rge

num

ber

of r

espo

nden

ts w

asin

com

plet

e.U

sefu

l res

pons

es 4

1% a

nd 6

8%Se

lf-ra

ted

heal

th in

Lith

uani

a an

d T

he N

ethe

rlan

ds

91N

icho

ls e

t al

.,19

96C

ronb

ach’

s α

= 0

.81

Can

cer

dete

ctio

n an

d pu

blic

ed

ucat

ion

88W

ilbur

et

al.,

1995

63%

res

pons

e ra

teC

ronb

ach’

s α

= 0

.96

Stud

y of

att

itude

s et

c to

war

ds

the

men

opau

se

87Va

lois

& G

odin

,199

1C

ronb

ach’

s α

for

4 ar

eas

such

as

Met

hods

pap

er o

n SD

Tsm

okin

g w

ere:

0.67

,0.

83,0

.49,

0.61

90W

ikbl

ad e

t al

.,19

9050

out

of 6

2 (8

1%)

poss

ible

res

pond

ents

Cro

nbac

h’s

α=

0.9

6H

igh

test

–ret

est

com

plet

ed t

echn

ique

cor

rect

lyT

he p

atie

nt’s

expe

rien

ce o

f R

elia

bilit

y co

effic

ient

diab

etes

and

tre

atm

ent

0.93


111Gut

tman

sca

les

Prop

erti

es: G

uttm

an s

cale

s h

ave

a th

eore

tica

l bas

is in

Fac

et T

heo

ry.98

No

info

rmat

ion

con

cern

ing

con

stra

ined

ch

oice

or

stre

ngt

h o

f pr

efer

ence

is p

rovi

ded.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

95K

ello

way

& B

arlin

g,19

93Lo

w r

espo

nse

rate

(1

stud

y ha

d r

= 0

.91

Test

–ret

est

relia

bilit

ySc

ale

posi

tivel

y re

late

d to

mea

sure

sre

spon

se r

ate

< 1

0%)

(6 m

onth

s ha

dof

uni

on lo

yalty

,will

ingn

ess

to w

ork

Stud

ies

of t

rade

uni

on a

ctiv

ities

C

oeffi

cien

t of

corr

elat

ion

of 0

.70;

for

the

unio

n,an

d sa

tisfa

ctio

n fr

omof

mem

bers

in C

anad

asc

alab

ility

= 0

.60

p<

0.0

1th

e un

ion

Con

sist

ent

with

the

aut

hors

’ hy

poth

eses

,ext

rins

ic a

nd in

trin

sic

job

satis

fact

ion

was

neg

ativ

ely

corr

elat

ed w

ith m

embe

rs’

part

icip

atio

n in

uni

on a

ctiv

ities

98Ed

mun

dson

et

al.,

1993

Con

veni

ence

sam

ple

r=

0.9

1

Util

ity o

f Gut

tman

sca

le fo

r te

chni

que

deve

lopm

ent

and

valid

atio

n

99Sa

ntos

& B

ooth

,199

6O

nly

66%

of r

etur

ned

ques

tionn

aire

s w

ere

usef

ul (

no d

ata

on h

ow a

ny d

istr

ibut

ed)

Stud

y m

eat

avoi

danc

e am

ong

stud

ents

Res

pons

e ra

te m

ax.<

66%

100

Sann

er,1

994

Res

pons

e ra

te o

f 65%

Stud

y of

pub

lic a

ttitu

de

tow

ards

org

an d

onat

ion

102

Pete

rson

,198

9R

espo

nse

rate

pos

tal q

uest

ionn

aire

:71%

Attit

udes

:C

oeffi

cien

t of

rep

ro-

Stud

y of

den

tal h

ealth

du

cibi

lity

= 0

.91

attit

udes

and

kno

wle

dge

Coe

ffici

ent

of

scal

abili

ty =

0.7

2

Know

ledg

e:C

oeffi

cien

t of

rep

ro-

duci

bilit

y =

0.8

2C

oeffi

cien

t of

sc

alab

ility

= 0

.27

r =

Coe

fficie

nt o

f rep

rodu

cibilit

y

Appendix 2

112 Ser

vice

qua

lity

(SE

RV

QU

AL

)Pr

oper

ties

: is

base

d in

mul

tipl

e di

scre

pan

cy t

heo

ry, a

nd

prov

ides

an

ord

inal

mea

sure

of

qual

ity.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

167

Scar

dina

,199

4C

onve

nien

ce s

ampl

e of

10:

5 th

ough

t th

eEs

tabl

ishe

d fo

r 4

ofsu

rvey

ade

quat

e an

d 5

thou

ght

it to

o lo

ng.

the

5 at

trib

utes

with

Patie

nt s

atis

fact

ion

with

R

epor

ted

no d

iffic

ulty

rea

ding

or

unde

rsta

ndin

gC

ronb

ach’

s α

rang

ing

nurs

ing

care

;als

o,bu

dget

pie

th

e in

stru

ctio

ns o

r st

atem

ents

from

0.7

4 to

0.9

8.of

100

poi

nts

divi

ded

betw

een

Oth

er a

ttri

bute

the

5 su

bsca

les

to d

eter

min

e em

path

y pe

rcep

tion

rela

tive

impo

rtan

ce(s

core

) =

0.4

0

168

Sarg

eant

& K

aehl

er,1

998

182

pers

onal

inte

rvie

ws

One

que

stio

n w

asid

entic

al b

ut n

ega-

Patie

nts’

sat

isfa

ctio

n w

ith

tivel

y ph

rase

d.66

%G

P se

rvic

esw

ere

perf

ectly

con

sist

-en

t an

d 90

% w

ere

with

in 1

Lik

ert

poin

t

442

Osw

ald

et a

l.,19

9847

2 co

mpl

eted

and

use

able

res

pons

es t

oU

sing

a q

uest

ionn

aire

with

13

item

s66

0 po

stal

sur

veys

(72

%)

take

n fr

om H

ospi

tal Q

ualit

y Tr

ends

* ,H

ospi

tal s

atis

fact

ion

– ai

m o

f th

e au

thor

s fo

und

that

“an

alys

isde

term

inin

g w

heth

er d

imen

sion

s su

gges

ts t

he d

imen

sion

s of

ot

her

than

tho

se u

sed

by

perc

eive

d qu

ality

for

the

stud

y SE

RVQ

UA

L w

ere

impo

rtan

tsa

mpl

e di

ffere

d fr

om t

hose

m

easu

red

by S

ERV

QU

AL”

171

Yous

sef e

t al

.,19

9617

4 pa

tient

s.R

espo

nse

rate

s w

ere

29%

for

The

aut

hors

con

clud

e th

at “

The

post

al q

uest

ionn

aire

s:80

% fo

r th

ose

hand

edSE

RVQ

UA

L in

stru

men

t its

elf s

houl

dEm

piri

cal s

tudy

usi

ng

out

at h

ospi

tals

and

36%

whe

n gi

ven

out

be s

ubje

ct t

o co

ntin

uous

ref

inem

ent

SERV

QU

AL

(on

a 9-

poin

t by

GPs

as it

s ap

plic

atio

n in

the

NH

S gr

ows”

scal

e) t

o m

easu

re p

atie

nts’

sa

tisfa

ctio

n w

ith h

ospi

tals

’ qu

ality

of s

ervi

ce

* H

ospi

tal Q

ualit

y Tre

nds

is a

trade

mar

k su

rvey

of h

ospi

tal c

orpo

ratio

ns o

f Am

erica

cont

inue

d


113Ser

vice

qua

lity

(SE

RV

QU

AL

) co

ntd

Prop

erti

es: i

s ba

sed

in m

ulti

ple

disc

repa

ncy

th

eory

, an

d pr

ovid

es a

n o

rdin

al m

easu

re o

f qu

alit

y.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

173

Hea

dley

& M

iller

,199

3O

f96

7 in

itial

mai

lings

,244

com

plet

ed t

heC

ronb

ach’

s α

= 0

.87

Tent

ativ

e co

nver

gent

val

idity

‘pre

-enc

ount

er’ s

urve

y an

d 15

9 of

the

se(s

ubsc

ales

ran

ged

repo

rted

in c

ompa

riso

n of

Satis

fact

ion

with

med

ical

co

mpl

eted

the

‘pos

t-en

coun

ter’

sur

vey

from

0.5

8 to

0.7

7)SE

RVQ

UA

L ite

ms

and

a un

iver

sal

serv

ices

in a

pri

mar

y ca

re s

ettin

gqu

ality

judg

emen

t

SERV

QU

AL

appr

opri

ate

for

heal

thca

re,b

ut r

esea

rche

rs s

houl

d w

atch

for

situ

atio

ns t

hat

call

for

adap

tatio

n

176

Rei

denb

ach

& S

andi

fer-

73%

res

pons

e ra

te fr

om t

elep

hone

sur

vey

Cro

nbac

h’s

αfo

rSm

allw

ood,

1990

to 3

00 p

atie

nts

fact

ors

rang

ed fr

om0.

83 t

o 0.

96M

odifi

ed S

ERV

QU

AL

(5-p

oint

sc

ale)

to

asse

ss p

atie

nt

perc

eptio

n of

ser

vice

qua

lity

in in

patie

nt,o

utpa

tient

and

em

erge

ncy

serv

ices

177

Lam

,199

783

/84

ques

tionn

aire

s w

ere

com

plet

ed a

nd

Cro

nbac

h’s

αof

5-

poin

t Li

kert

sca

le r

athe

r th

an t

heon

ly 1

of t

hese

was

unu

sabl

e0.

64–0

.88

reco

mm

ende

d 7-

poin

t us

ed s

ince

Met

hodo

logi

cal/e

mpi

rica

l stu

dy

pilo

ting

indi

cate

d th

is w

ould

red

uce

usin

g SE

RVQ

UA

L to

det

erm

ine

frus

trat

ion

and

incr

ease

res

pons

epa

tient

s’ p

erce

ptio

n of

the

ra

te a

nd q

ualit

y of

res

pons

es.I

t al

soqu

ality

of h

ealth

care

in

led

to t

he r

emov

al o

f neg

ativ

ely

Hon

g K

ong

wor

ded

stat

emen

ts t

hat

had

led

to c

onfu

sion

and

irri

tatio

n fo

r th

e re

spon

dent

s

178

Duf

fy e

t al

.,19

9720

6 U

SA a

nd 1

00 U

K p

artic

ipan

tsC

ronb

ach’

s α

= 0

.95

See

‘Val

idity

’R

elia

bilit

y an

d va

lidity

of S

ERV

QU

AL

and

0.88

res

pect

ivel

y,ha

s be

en e

stab

lishe

d in

non

-hea

lth2-

cent

re (

USA

and

UK

) A

utho

rs n

ote

inte

rvie

win

g to

ok 1

–1.5

hou

rsal

thou

gh 2

sub

scal

elit

erat

ure

empi

rica

l stu

dy o

f nur

sing

ra

tings

in t

he U

Kho

me

resi

dent

s’ e

valu

atio

n sa

mpl

e w

ere

~ 0

.4of

ser

vice

qua

lity

cont

inue

d

Appendix 2

114 Ser

vice

qua

lity

(SE

RV

QU

AL

) co

ntd

Prop

erti

es: i

s ba

sed

in m

ulti

ple

disc

repa

ncy

th

eory

, an

d pr

ovid

es a

n o

rdin

al m

easu

re o

f qu

alit

y.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

180

Mitc

hell

et a

l.,19

9943

% r

espo

nse

rate

(86

/200

)C

ronb

ach’

s α

=C

onte

nt v

alid

ity w

as e

stab

lishe

d0.

81–0

.96

for

the

thro

ugh

qual

itativ

e st

udie

sU

sing

mod

ified

SER

VQ

UA

L 5

subi

tem

s(5

-poi

nt s

cale

) to

ass

ess

empl

oyee

s’ s

atis

fact

ion

with

th

e qu

ality

of n

ursi

ng s

ervi

ces.

Min

or w

ordi

ng m

odifi

catio

ns

370

Baba

kus

& M

ango

ld,1

992

A 5

-poi

nt L

iker

t sc

ale

was

use

d in

stea

d of

the

Cro

nbac

h’s

αfo

rEv

iden

ce o

f cor

rela

tion

betw

een

orig

inal

7-p

oint

.22%

res

pons

e ra

te fr

om a

subs

cale

s ra

nged

from

expe

ctat

ions

and

per

cept

ions

Eval

uatin

g th

e ap

plic

atio

n of

po

stal

que

stio

nnai

re (

443/

1999

)0.

759

to 0

.903

SERV

QU

AL

to h

ospi

tal s

ervi

ces

182

Duf

fy &

Ket

chan

d,19

9820

6 nu

rsin

g ho

me

resi

dent

s (m

ean

age

79)

Cro

nbac

h’s

α=

0.9

5U

nexp

lain

ed v

aria

tion

(55%

) in

fers

wer

e in

clud

ed;n

o m

entio

n of

res

pons

e ra

teth

at d

imen

sion

s w

hich

may

hav

eU

se o

f SER

VQ

UA

L to

ass

ess

influ

ence

d sa

tisfa

ctio

n w

ere

excl

uded

patie

nt s

atis

fact

ion

with

nur

sing

ho

me

care

447

Har

t,19

967-

poin

t Li

kert

sca

le m

ay n

ot h

ave

equa

l int

erva

ls a

nd r

espo

nden

ts

Empi

rica

l/met

hodo

logi

cal s

tudy

m

ay “

actu

ally

dep

loy

an ‘i

ncre

asin

gre

port

ing

on 4

stu

dies

on

resi

stan

ce’ m

odel

in w

hich

it is

eas

ier

outp

atie

nt c

linic

s w

hich

hav

e (in

psy

chom

etri

c te

rms)

to

mov

eus

ed S

ERV

QU

AL

to m

easu

re

from

the

cen

tral

poi

nt (

poin

t 4)

to

qual

ity.5

dim

ensi

ons

wei

ghte

d its

imm

edia

te n

eigh

bour

(po

int

5)to

sum

100

than

it is

to

mov

e fr

om a

pos

ition

of

nea

r pe

rfec

t sa

tisfa

ctio

n (p

oint

6)

to p

erfe

ct s

atis

fact

ion

(poi

nt 7

)”.

Inst

ead,

the

auth

or s

ugge

sts

assi

gnin

g ca

rdin

al v

alue

s to

eac

h po

int

as

follo

ws:

poin

t on

sca

le 1

(va

lue

–6);

2 (–

3);3

(–1

);4

(0);

5 (1

);6

(3);

7 (6

)

445

Tom

es &

Ng,

1995

The

49-

item

sca

le

Dim

ensi

ons

wer

e no

t ta

ken

from

had

good

inte

rnal

SERV

QU

AL

but

wer

e de

velo

ped

Dev

elop

men

t of

a q

uest

ionn

aire

,co

nsis

tenc

y (0

.959

);af

ter

hosp

ital s

taff

inte

rvie

ws

base

d on

SER

VQ

UA

L,to

in

divi

dual

dim

ensi

ons

mea

sure

hos

pita

l ser

vice

qua

lity

also

rev

eale

d hi

gh

cons

iste

ncie

s


115Sim

ple

cho

ice

exer

cise

s an

d ra

ndo

m p

aire

d co

mpa

riso

ns

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

184

Lew

is &

Cha

rny,

1999

Res

pond

ents

favo

ured

the

5-y

ear-

old

over

the

70-

year

-old

by

a ra

tio o

fSu

bset

of t

he C

ardi

ff st

udy18

384

:1 a

nd t

he 3

5-ye

ar-o

ld t

o th

ede

alin

g w

ith t

he r

elat

ive

valu

e 60

-yea

r-ol

d by

14:

1,bu

t fa

vour

ed

of li

fe d

epen

dent

on

age

the

8-ye

ar-o

ld o

ver

the

2-ye

ar-o

ld

com

pari

sons

onl

yby

5:3

(se

e be

low

).H

ighl

ight

s co

mpl

exity

of a

ge-b

ased

cho

ices

183

Cha

rny

et a

l.,19

89O

f the

722

res

pond

ents

,220

(31

%)

mad

e al

lA

s in

ear

lier

findi

ngs,

resu

lts fa

vour

ed13

cho

ices

,whi

lst

near

ly 7

5% m

ade

choi

ces

youn

g ov

er o

ld,m

arri

ed o

ver

sing

le,

Car

diff

stud

y el

iciti

ng

in h

alf o

r m

ore

of t

he 1

3 qu

estio

ns.T

he fa

ctw

omen

ove

r m

en a

nd n

on-s

mok

ers

pref

eren

ces

of 1

spe

cific

life

th

at 6

9% w

ere

unab

le t

o m

ake

all t

he c

hoic

es,

and

drin

kers

ove

r sm

oker

s an

dre

lativ

e to

ano

ther

.Res

pond

ents

in

dica

tes,

in t

he o

pini

on o

f the

aut

hors

,tha

tdr

inke

rs.T

he in

stan

ce w

here

an

had

to c

hoos

e be

twee

n 2

resp

onde

nts

took

the

exe

rcis

e ve

ry s

erio

usly

8-ye

ar-o

ld c

hild

was

favo

ured

ove

r hy

poth

etic

al in

divi

dual

s w

ho

a 2-

year

-old

was

exp

lain

ed b

y th

edi

ffere

d by

onl

y 1

char

acte

rist

ic

grea

ter

pare

ntal

inve

stm

ent

in t

erm

s(a

ge,m

arita

l sta

tus,

gend

er,

of e

ffort

and

em

otio

nsm

okin

g/dr

inki

ng s

tatu

s,an

d em

ploy

men

t)

186

Ryy

nane

n et

al.,

1996

Aut

hors

not

e th

at m

embe

rs o

f the

pub

licTe

st–r

etes

t re

liabi

lity

Cri

teri

on v

alid

ity w

as e

xam

ined

(n

= 4

9) c

ompl

eted

the

que

stio

nnai

res

quic

kly;

was

mea

sure

d by

by c

ompa

ring

the

und

ergr

adua

teM

etho

dolo

gica

l/em

piri

cal s

tudy

ho

wev

er,‘m

ost’

of t

he p

ublic

com

plet

ing

the

8 re

spon

dent

sst

uden

ts’ r

espo

nses

to

the

RPS

of p

rior

itisa

tion

betw

een

a ex

erci

se fo

und

diffi

culty

in m

akin

g ch

oice

san

swer

ing

3 di

ffere

ntqu

estio

ns c

ompa

red

with

num

ber

of ‘e

thic

al v

alue

be

twee

n 2

alte

rnat

ives

.Som

e fo

und

that

the

sets

of R

PS q

uest

ions

.co

nven

tiona

l que

stio

nnai

res,

and

indi

cato

rs’ (

cons

istin

g of

age

,qu

estio

nnai

re m

ade

them

anx

ious

and

1A

utho

rs s

tate

tha

tco

mpa

ring

a n

on-s

truc

ture

din

com

e,se

veri

ty o

f dis

ease

,re

acte

d ag

gres

sive

ly.T

he m

edic

al a

nd n

ursi

ngre

liabi

lity

was

“go

od”

inte

rvie

w w

ith t

he R

PS fo

r pr

ogno

sis,

soci

al s

tatu

s,co

st

unde

rgra

duat

es (

n=

104

) co

mpl

eted

the

mem

bers

of t

he p

ublic

;the

of

tre

atm

ent,

and

orig

in

ques

tions

qui

ckly

and

with

out

diffi

culty

resu

lts w

ere

com

para

ble

of d

isea

se)

Appendix 2

116 CA

cho

ice-

base

d qu

esti

ons

Prop

erti

es: c

hoi

ce-b

ased

CA

is r

oote

d in

ran

dom

uti

lity

theo

ry60

,61,

193,

198,

201,

203

and

Lan

cast

er’s

th

eory

of

valu

e.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

187

Brya

n et

al.,

1998

Con

veni

ence

sam

ple

of 1

34 s

tude

nts:

The

oret

ical

val

idity

was

sup

port

ed

all r

espo

nded

with

a p

riori

hypo

thes

is c

onfir

med

Pref

eren

ces

for

use

of

mag

netic

res

onan

ce im

agin

g 82

(61

%)

resp

onde

nts

wer

e no

tin

kne

e in

juri

esw

illin

g to

tra

de

188

Cha

krab

orty

et

al.,

1994

662

resp

onde

nts

part

icip

ated

from

777

appr

oach

ed.E

ach

was

pai

d $1

5 fo

rEm

piri

cal s

tudy

usi

ng C

A t

o pa

rtic

ipat

ion.

Post

al q

uest

ionn

aire

eval

uate

hea

lth in

sura

nce

prog

ram

mes

(4

alte

rnat

ives

,ea

ch w

ith 2

4 at

trib

utes

)

189

Farr

ar &

Rya

n,19

99T

heor

etic

al v

alid

ity w

as s

uppo

rted

with

a p

riori

hypo

thes

is c

onfir

med

Con

sulta

nt p

refe

renc

es fo

r al

tern

ativ

e cl

inic

al s

ervi

ce

The

aut

hors

foun

d no

evi

denc

e of

deve

lopm

ents

orde

ring

effe

cts

in t

erm

s of

ord

erin

g of

att

ribu

tes

with

in s

cena

rios

191

Ferg

uson

et

al.,

1994

44 p

atie

nts,

mea

n ag

e 75

yea

rs.2

1/24

agr

eed

to p

artic

ipat

e in

a c

linic

gro

up a

nd 2

5/28

inPi

lot

stud

y of

eld

erly

pat

ient

s’

an o

ffice

gro

up.R

espo

nse

rate

= 8

8%.

attit

udes

ove

r ch

oice

of d

rugs

2

patie

nts

had

diffi

culty

und

erst

andi

ng t

hefo

r tr

eatm

ent

of h

yper

tens

ion:

ques

tionn

aire

and

wer

e ex

clud

ed3

attr

ibut

es p

rese

nted

in

pair

s on

ly

619

Free

man

et

al.,

1998

The

aut

hors

not

e th

e m

etho

d is

sim

pler

th

an r

anki

ng a

nd r

atin

g C

A m

etho

ds a

ndSt

uden

t pr

efer

ence

s fo

r re

spon

dent

fatig

ue is

red

uced

.Als

o,le

vel

heal

thca

re n

eeds

usi

ng C

A

of a

ccur

acy

is in

crea

sed

(max

imum

diff

eren

ce

CA

met

hod)

192

Hak

im &

Pat

hak,

1999

Evid

ence

of c

onve

rgen

t va

lidity

be

twee

n ch

oice

-bas

ed C

A,r

atin

g C

ompa

riso

n of

Eur

oqol

sc

ales

and

SG

(Eur

opea

n qu

ality

of l

ife)

heal

th-

stat

e pr

efer

ence

s us

ing

choi

ce-

base

d C

A,r

atin

g sc

ales

and

SG

cont

inue

d


117CA

cho

ice-

base

d qu

esti

ons

con

td

Prop

erti

es: c

hoi

ce-b

ased

CA

is r

oote

d in

ran

dom

uti

lity

theo

ry60

,61,

193,

198,

201,

203

and

Lan

cast

er’s

th

eory

of

valu

e.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

193

McI

ntos

h &

Rya

n,19

99R

espo

nse

rate

was

56%

(55

6/10

00)

6% o

f res

pond

ents

The

oret

ical

val

idity

was

sup

port

edfo

r a

post

al q

uest

ionn

aire

prov

ided

intr

ansi

tive

with

a p

riori

hypo

thes

is c

onfir

med

Pref

eren

ces

for

loca

tion

resp

onse

sof

tre

atm

ent

63%

of t

he s

ampl

e vi

olat

ed t

heco

ntin

uity

axi

om,i

.e.t

hey

show

ed

dom

inan

t pr

efer

ence

s

195

Prop

per,

1990

The

oret

ical

val

idity

was

sup

port

edw

ith a

prio

rihy

poth

esis

con

firm

edEs

timat

ion

of t

he v

alue

of

redu

cing

wai

ting

time

The

aut

hor

note

s th

e pr

esen

ce o

f no

n-tr

adin

g (o

r le

xico

grap

hic)

pr

efer

ence

s in

4%

196

Rat

cliff

e &

Bux

ton,

1999

Res

pons

e ra

te w

as 8

9% (

189/

213)

usi

ng p

osta

l9%

of r

espo

nden

tsSt

udy

used

leve

ls o

f att

ribu

tes

that

ques

tionn

aire

,inc

ludi

ng c

over

ing

lett

er fr

omga

ve in

cons

iste

ntm

ost

clos

ely

rese

mbl

ed e

xist

ing

Patie

nt p

refe

renc

es fo

r ph

ysic

ian

and

with

1 r

emin

der.

59%

(11

1)an

swer

sse

rvic

es o

n of

fer.

Inte

rnal

val

idity

outc

ome

and

proc

ess

foun

d th

e qu

estio

nnai

re w

as n

ot d

iffic

ult

toco

nfir

med

attr

ibut

es in

live

r co

mpl

ete

and

26%

(48

) th

ough

t it

slig

htly

tran

spla

ntat

ion

diffi

cult.

Mea

n tim

e fo

r co

mpl

etio

n of

the

qu

estio

nnai

re w

as 1

6 m

inut

es (

rang

e 10

–60)

197

Rya

n,19

9933

1/41

4 qu

estio

nnai

res

wer

e co

mpl

eted

:Lo

w le

vels

of

The

oret

ical

val

idity

was

sup

port

ed59

non

-res

pons

esin

cons

iste

ncy:

first

with

a p

riori

hypo

thes

is c

onfir

med

Empi

rica

l/met

hodo

logi

cal

ques

tionn

aire

= 4

and

appl

icat

ion

of C

A t

o IV

F se

cond

= 6

The

aut

hor

foun

d su

ppor

t fo

rtr

eatm

ent

inte

rnal

val

idity

in t

erm

s of

a p

riori

expe

ctat

ions

All

resp

onde

nts

wer

e tr

ader

s

205

Rya

n &

Far

rar,

2000

157/

160

ques

tionn

aire

s w

ere

com

plet

edA

utho

rs n

ote

the

8 re

spon

dent

sT

heor

etic

al v

alid

ity w

as s

uppo

rted

te

chni

que

was

answ

ered

at

leas

t 1

with

a p

riori

hypo

thes

is c

onfir

med

App

licat

ion

of C

A t

o w

ell r

ecei

ved

byof

cho

ices

2 o

r 13

orth

odon

tic s

ervi

ces

(loca

tion,

polic

y-m

aker

sin

cons

iste

ntly

and

37

wai

ting

time)

:5-p

oint

sca

lean

swer

ed c

hoic

e 10

in

cons

iste

ntly

cont

inue

d

Appendix 2

118 CA

cho

ice-

base

d qu

esti

ons

con

td

Prop

erti

es: c

hoi

ce-b

ased

CA

is r

oote

d in

ran

dom

uti

lity

theo

ry60

,61,

193,

198,

201,

203

and

Lan

cast

er’s

th

eory

of

valu

e.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

199

Rya

n &

Hug

hes,

1997

33%

res

pons

e ra

te (

196)

.85

resp

onde

nts

Res

pond

ents

gav

e on

lyEv

iden

ce w

as fo

und

to s

uppo

rtth

ough

t it

easy

or

extr

emel

y ea

sy,4

5 di

fficu

lta

smal

l num

ber

ofth

eore

tical

val

idity

Wom

en’s

pref

eren

ces

for

and

9 ex

trem

ely

diffi

cult

inco

nsis

tent

ans

wer

sm

isca

rria

ge m

anag

emen

t32

res

pond

ents

con

sist

ently

had

a

pref

eren

ce fo

r 1

type

of t

reat

men

t ov

er a

noth

er,a

nd w

ere

not

will

ing

to t

rade

200

Rya

n &

Wor

dsw

orth

,200

0A

pilo

t fo

und

13 c

hoic

es t

o be

too

man

ySu

ppor

t fo

r th

eore

tical

val

idity

Wom

en’s

pref

eren

ces

for

641

usab

le r

espo

nses

wer

e re

ceiv

ed fr

omLe

vels

of a

ttri

bute

s w

ere

vari

ed,

cerv

ical

scr

eeni

ng p

rogr

amm

es20

00 a

fter

2 r

emin

ders

whi

lst

the

coef

ficie

nts

wer

e no

t si

gnifi

cant

ly d

iffer

ent

acro

ss 5

of

the

6 at

trib

utes

;mea

n W

TP

was

si

gnifi

cant

ly d

iffer

ent

for

4 of

the

5

wel

fare

est

imat

es

201

San

Mig

uel e

t al

.,20

01C

onve

nien

ce s

ampl

e of

146

wom

en.N

oSu

ppor

t fo

r th

eore

tical

val

idity

rem

inde

rs.R

espo

nse

rate

was

51%

(75

/146

).Pr

efer

ence

s fo

r of

men

orrh

agia

Res

pond

ents

foun

d qu

estio

nnai

re n

ot t

ooEv

iden

ce o

f lim

ited

trad

ing,

with

diffi

cult,

alth

ough

the

re w

as a

spr

ead

of

36 (

48%

) re

spon

dent

s co

nsis

tent

lyre

port

ed d

iffic

ulty

.Tim

e fo

r co

mpl

etio

n ha

ving

a p

refe

renc

e fo

r 1

type

of

was

14

min

utes

on

aver

age

trea

tmen

t ov

er a

noth

er

357

San

Mig

uel e

t al

.,19

9954

% r

espo

nse

rate

(73

5/13

64):

731

Kap

pa c

oeffi

cien

t of

wer

e us

able

0.69

and

0.6

8Te

stin

g fo

r co

mpl

eten

ess

and

stab

ility

pre

fere

nces

for

out-

of-

hour

s ca

re

202

Scot

t &

Vic

k,19

9918

.4%

res

pons

e ra

te (

734/

3983

) po

stal

97%

of r

espo

nses

Pr

ior

hypo

thes

es w

ere

conf

irm

ed,

ques

tionn

aire

s (n

o re

min

ders

).82

% fo

und

cons

iste

ntin

dica

ting

theo

retic

al v

alid

ity (

i.e.

Patie

nt p

refe

renc

es fo

r th

e qu

estio

nnai

re o

kay

or e

asy

to c

ompl

ete

patie

nts

pref

erre

d m

ore

info

rmat

ion

attr

ibut

es o

f the

doc

tor–

whi

lst

18%

foun

d it

diffi

cult

or e

xtre

mel

yto

less

,pre

fer

easy

exp

lana

tions

pa

tient

rel

atio

nshi

pdi

fficu

lt.A

utho

rs n

ote

that

due

to

the

low

fr

om t

he d

octo

r,pr

efer

the

doc

tor

resp

onse

rat

e,th

ose

retu

rnin

g th

e qu

estio

n-lis

teni

ng t

o w

hat

they

hav

e to

say

,na

ire

may

hav

e be

en t

he o

nes

who

foun

d an

d pr

efer

a s

hort

er w

aitin

g tim

e)it

easy

cont

inue

d


119CA

cho

ice-

base

d qu

esti

ons

con

td

Prop

erti

es: c

hoi

ce-b

ased

CA

is r

oote

d in

ran

dom

uti

lity

theo

ry60

,61,

193,

198,

201,

203

and

Lan

cast

er’s

th

eory

of

valu

e.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

206

Spot

h &

Red

mon

d,19

93U

se A

CA

(ad

aptiv

e C

A)

soft

war

e to

faci

litat

e0.

90 o

r gr

eate

r co

rrel

atio

n fo

r 86

%20

2 te

leph

one

inte

rvie

ws

(99.

5% r

espo

nse

of t

he s

ampl

e be

twee

n C

A r

esul

tsPa

tient

pre

fere

nces

for

rate

).O

f a 2

5-m

inut

e in

terv

iew

,the

CA

par

tan

d ra

ting

exer

cise

prev

entio

n pr

ogra

mm

esto

ok a

roun

d 15

min

utes

Met

hod

is s

imila

r to

tha

t fa

ced

in

real

-life

situ

atio

ns w

here

res

pond

ents

ch

oose

bet

wee

n di

ffere

nt

prog

ram

me

prof

iles

620

Szei

nbac

h et

al.,

1999

CA

and

VA

S re

sults

wer

eco

mpa

rabl

e,sh

owin

g co

nver

gent

Elic

iting

pat

ient

pre

fere

nces

for

valid

ityhe

alth

sta

tes

usin

g C

A a

nd V

AS

Sim

ilar

resp

onse

s fo

r VA

S an

d C

A in

2

prof

iles

with

sim

ilar

outc

omes

bu

t w

orde

d di

ffere

ntly

203

Vic

k &

Sco

tt,1

998

63%

res

pons

e ra

te (

101/

160)

;mea

n tim

e Fo

r th

e 1

dom

inan

tT

heor

etic

al v

alid

ity is

sup

port

ed fo

rof

com

plet

ion

abou

t 10

min

utes

;80%

of

optio

n,80

–90%

of

2 of

the

3 a

ttri

bute

s,bu

t co

st w

asR

elat

ive

impo

rtan

ce t

o re

spon

dent

s fo

und

the

ques

tionn

aire

oka

y to

resp

onse

s w

ere

not

sign

ifica

ntpa

tient

s of

the

att

ribu

tes

of

very

eas

y,w

hils

t 19

% fo

und

it qu

ite d

iffic

ult

cons

iste

ntG

P co

nsul

tatio

nan

d 2%

ver

y di

fficu

ltSo

me

evid

ence

of o

rder

ing

effe

cts

Appendix 2

120 Ana

lyti

c hi

erar

chy

pro

cess

(A

HP

)Pr

oper

ties

: Saa

ty’s

219

axio

ms

unde

rlyi

ng

the

theo

reti

cal f

oun

dati

on o

f th

e A

HP

are

not

ed22

7,23

2,23

3an

d th

e m

eth

od r

efle

cts

inte

nsi

ty o

f pr

efer

ence

.238

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

225

Schw

artz

& O

ren,

1988

83%

com

plet

ed t

he q

uest

ions

.28

pair

wis

e5%

of r

espo

nden

tsco

mpa

riso

ns n

ot s

een

as e

xces

sive

,and

wer

ew

ere

excl

uded

from

Patie

nt p

refe

renc

es fo

r a

new

ea

sily

und

erst

ood

the

anal

ysis

bec

ause

inte

rven

tion

for

reco

very

from

th

eir

cons

iste

ncy

myo

card

ial i

nfar

ctio

nra

tios

exce

eded

0.5

0 (t

he o

ther

s ra

nged

fr

om 0

.11

to 0

.23)

235

Dol

an,1

995

20 p

atie

nts

part

icip

ated

,18

(90%

) w

ere

Find

ings

are

con

sist

ent

with

pre

viou

sca

pabl

e (ju

dged

as

able

to

com

plet

e th

ere

sults

,232

alth

ough

thi

s st

udy

Stud

y se

ekin

g to

det

erm

ine

exer

cise

in ≤

45 m

inut

es)

and

will

ing

to g

oin

volv

ed s

mal

l num

bers

whe

ther

pat

ient

s ar

e w

illin

g th

roug

h A

HP

anal

ysis

bef

ore

mak

ing

aan

d ab

le t

o us

e th

e A

HP,

with

cl

inic

al d

ecis

ion

Valid

ity o

f AH

P ha

s co

ntes

ted

choi

ces

of a

ltern

ativ

e sc

reen

ing

thro

ugh

rank

rev

ersa

l (in

whi

ch a

regi

men

s fo

r co

lon

canc

er

Patie

nts

com

plet

ed t

he A

HP

exer

cise

on

a ch

ange

in r

elat

ive

desi

rabi

lity

isus

ed a

s an

exa

mpl

ela

ptop

com

pute

r.Fo

llow

ing

the

AH

P ex

erci

se,

caus

ed b

y th

e in

trod

uctio

n of

resp

onde

nts

wer

e as

ked

eval

uatio

n qu

estio

ns:

anot

her

alte

rnat

ive

into

the

ana

lysi

s).

15%

sai

d th

e ex

erci

se w

as h

ard

to u

nder

stan

d;N

ew m

etho

d of

AH

P pr

eclu

des

95%

lear

ned

usef

ul in

form

atio

n du

ring

the

ra

nk r

ever

sal

exer

cise

;90–

100%

sai

d th

ey w

ould

like

AH

P to

influ

ence

dec

isio

n-m

akin

g

226

Dol

an,1

989

Stud

y de

mon

stra

tes

the

usef

ulne

ss

of A

HP,

whi

ch le

d to

a c

hang

e in

Det

erm

ine

best

ant

ibio

tic

pres

crib

ing

beha

viou

r to

impr

ove

regi

men

for

wom

an w

ith

the

proc

ess

of p

atie

nt c

are

acut

e py

elon

ephr

itis

239

Java

lgi e

t al

.,19

8947

% (

235)

res

pons

e ra

te fr

om 5

00 p

osta

lqu

estio

nnai

res

(with

a $

1 in

cent

ive)

;a fu

rthe

rPr

esen

tatio

n of

the

AH

P an

d 15

of w

hich

wer

e un

usab

leill

ustr

atio

n ap

plie

d to

pub

lic

pref

eren

ces

for

hosp

ital

attr

ibut

es w

hen

sele

ctin

g a

hosp

ital f

or t

reat

men

t

cont

inue

d


121Ana

lyti

c hi

erar

chy

pro

cess

(A

HP

) co

ntd

Prop

erti

es: S

aaty

’s21

9ax

iom

s un

derl

yin

g th

e th

eore

tica

l fou

nda

tion

of

the

AH

P ar

e n

oted

227,

232,

233

and

the

met

hod

ref

lect

s in

ten

sity

of

pref

eren

ce.23

8

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

236

Dol

an,1

993

Prev

ious

wor

k ha

s de

mon

stra

ted

feas

ibili

tyA

utho

r no

tes

that

thi

s an

d pr

evio

usw

ith p

atie

nts,

453,

455

80%

of t

he fo

rmer

453

appl

icat

ions

218,

227,

228,

237,

621

dem

onst

rate

Illus

trat

ion

of t

he A

HP

for

ratin

g th

e A

HP

feas

ible

and

22

patie

nts

and

the

AH

P’s

suita

bilit

y fo

r m

edic

altr

eatm

ent

optio

ns fo

r a

man

25

/26

phys

icia

ns in

the

latt

er45

5su

cces

sful

lyde

cisi

on-m

akin

g an

d co

unte

ract

with

a c

omm

on s

erio

us d

isea

seco

mpl

etin

g th

e ex

erci

se w

ithin

a r

easo

nabl

e qu

estio

ning

by

Eckm

an62

2

amou

nt o

f tim

e

621

McN

eil e

t al

.,19

82R

espo

nden

ts a

ppea

red

to u

nder

stan

d th

eda

ta p

rese

nted

,alth

ough

the

y w

ere

influ

ence

dPa

tient

,stu

dent

and

phy

sici

an

by t

he n

atur

e of

the

dat

a an

d th

e fo

rm in

choi

ces

of a

ltern

ativ

e th

erap

ies

whi

ch it

was

pre

sent

edfo

r lu

ng c

ance

r us

ing

vari

atio

ns

of b

ackg

roun

d in

form

atio

n pr

ovid

ed

229

Dol

an &

Bor

dley

,199

4Va

lidity

of A

HP

is d

ebat

edco

ncer

ning

the

ran

k re

vers

alU

se o

f AH

P co

ncer

ning

ph

enom

enon

.Aut

hors

use

d a

ison

iazi

d pr

ophy

laxi

s in

m

etho

d of

agg

rega

tion

whe

re

posi

tive

tube

rcul

in t

ests

rank

rev

ersa

l can

not

occu

r

452

Pera

lta-C

arce

len

et a

l.,19

9790

% (

83/9

2) o

f pre

gnan

t w

omen

,51%

3

of t

he w

omen

(40/

78)

of o

bste

tric

ians

and

67%

(40

/60)

of

patie

nts

prov

ided

Empi

rica

l stu

dy o

f the

pa

edia

tric

ians

par

ticip

ated

.89%

(74

) of

wom

enin

cons

iste

nt r

espo

nses

pref

eren

ces

of p

regn

ant

wom

en,

and

90%

(72

) of

phy

sici

ans

unde

rsto

od t

hew

hils

t th

ere

wer

e no

phys

icia

ns a

nd p

aedi

atri

cian

s fo

r in

terv

iew

form

at.A

‘few

’ of t

he w

omen

pat

ient

sin

cons

iste

ncie

s w

ithre

duci

ng in

cide

nce

of n

eona

tal

had

diffi

culty

und

erst

andi

ng s

ome

of t

heth

e ph

ysic

ians

grou

p B

stre

ptoc

occa

l sep

sis

crite

ria,

as d

id s

ome

of t

he p

hysi

cian

s.87

% o

fus

ing

AH

Pw

omen

pat

ient

s,88

% o

f pae

diat

rici

ans

and

75%

of

obs

tetr

icia

ns li

ked

the

inte

rvie

w fo

rmat

.C

once

rn e

xpre

ssed

tha

t w

omen

with

lim

ited

educ

atio

nal b

ackg

roun

ds m

ight

hav

e di

fficu

lty

unde

rsta

ndin

g th

e m

odel

;how

ever

,no

evid

ence

of t

his.

Giv

en t

he m

odes

t ed

ucat

iona

l ba

ckgr

ound

of t

he s

ampl

e,th

e au

thor

s be

lieve

th

e m

etho

d to

be

gene

ralis

able

Appendix 2

122 Sta

ndar

d ga

mbl

e (S

G)

Prop

erti

es: S

G is

roo

ted

in E

UT

245,

352,

456,

457

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.45

8

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

253

Braz

ier

et a

l.,19

99SG

Giv

en t

hat

SGLl

ewel

lyn-

Tho

mas

Frob

erg

and

Kan

e352

SG b

ased

on

EUT

dev

elop

ed b

y Em

piri

cal s

tudi

es u

sing

SG

hav

e re

port

edqu

estio

ns a

reet

al.3

61re

port

54

of

pres

ent

evid

ence

of

von

Neu

man

n an

d M

orge

nste

rn††

.R

esul

ts o

f a s

yste

mat

ic r

evie

w

resp

onse

and

com

plet

ion

rate

s to

co

llect

ed b

y64

(84

%)

resp

onde

nts

good

inte

r-ra

ter

Giv

en t

his,

it is

oft

en v

iew

ed a

s th

eof

the

use

of h

ealth

sta

tus

dem

onst

rate

a h

igh

degr

ee o

f acc

epta

nce.

trai

ned

inte

r-ra

nked

5 h

ealth

sta

tes

relia

bilit

y (r

= 0

.77;

clas

sic

met

hod

of d

ecis

ion-

mak

ing

mea

sure

s in

eco

nom

ic

Man

y st

udie

s ha

ve r

epor

ted

com

plet

ion

rate

svi

ewer

s,th

ew

ith a

prio

rifr

om T

orra

nce40

0an

dun

der

unce

rtai

nty.24

5

eval

uatio

nbe

twee

n 80

% a

nd 1

00%

361,

411–

413,

459–

462

tech

niqu

e w

ill

expe

ctat

ions

test

–ret

est

relia

bilit

ybe

rel

ativ

ely

(r=

0.8

0))

How

ever

,the

re is

a s

ubst

antia

l bod

yT

he a

utho

r ci

tes

that

“al

thou

gh S

G h

as s

how

n ex

pens

ive35

2,40

0Le

nert

et

al.41

1fo

und

of e

vide

nce

show

ing

that

indi

vidu

als

som

e co

mpl

etio

n pr

oble

ms

with

in p

artic

ular

78

% o

f hea

lthy

subj

ects

The

follo

win

g de

tails

cons

iste

ntly

vio

late

the

axi

oms

stud

ies,

thes

e ha

ve b

een

no w

orse

tha

n si

mila

r an

d 44

% o

f pat

ient

sm

ore

rece

nt s

tudi

esof

EU

Tdi

fficu

lties

ass

ocia

ted

with

oth

er in

stru

men

ts

had

a co

nsis

tent

ran

kid

entif

ied

by B

razi

erus

ed a

t th

e sa

me

time”

.For

exa

mpl

e,w

here

or

deri

ng o

f pre

fer-

et a

l.:SG

val

uatio

ns m

ay b

e af

fect

ed b

yco

mpl

etio

n pr

oble

ms

occu

rred

in s

tudi

es b

y en

ces

amon

g VA

S an

d1

wee

k or

less

= 0

.8* ;

diffe

ring

att

itude

s to

ris

k,i.e

.whe

ther

Patr

ick

et a

l.463

and

van

der

Don

k et

al.,

414

pair

wis

e co

mpa

riso

n,0.

77–0

.79

†pe

ople

are

ris

k-av

erse

,ris

k-ne

utra

lw

hich

use

d SG

,TTO

and

VA

S m

etho

ds,S

G

whi

lst

only

80%

and

4 w

eeks

:0.8

2‡

or r

isk-

seek

ing.

Loom

es a

ndw

as n

ot s

een

to b

e m

ore

burd

enso

me

than

61

%,r

espe

ctiv

ely,

had

6–16

wee

ks:

McK

enzi

e623

sugg

est

that

at

times

othe

r m

etho

ds e

mpl

oyed

a co

nsis

tent

ran

k0.

63 p

rops

§ ;pe

ople

may

be

a m

ixtu

re o

f all

3or

deri

ng o

f pre

fer-

0.74

no

prop

s§

Llew

elly

n-T

hom

as e

t al

.361

stat

e th

at t

heen

ces

amon

g SG

and

1

year

:0.5

3#Ev

iden

ce s

uppo

rts

the

conv

erge

ntSG

is a

com

plex

met

hod

pair

wis

e co

mpa

riso

n ot

her:

0.82

¶ va

lidity

of S

G w

ith T

TO46

2,46

7,46

8,46

9

ratin

gstim

e un

spec

ified

:0.8

0**an

d W

TP80

,324

Intr

acla

ss c

orre

latio

n SG

doe

s no

t co

rrel

ate

wel

l with

coef

ficie

nt†,

‡,**;

eith

er h

ealth

sta

tus

or V

AS.

409,

422

Pear

son

corr

elat

ion

SG v

alue

s ha

ve g

ener

ally

bee

n co

effic

ient

§ ;oth

ers

foun

d to

exc

eed

thos

e of

VA

S un

spec

ified

met

hods

409,

412,

413,

414,

419,

459,

469

* O

’Con

nor41

8 ;† Ba

kker

et

al.40

9 ;‡ O

’Brie

n an

d Vi

ram

onte

s419 ;§

Dol

an e

t al

.462 ;#

Torr

ance

400 ;¶

Reed

et

al.46

6 ;** G

age

et a

l.459

†† Vo

n N

eum

ann

J,M

orge

nste

rn O

.The

ory

of g

ames

and

eco

nom

ic be

havio

r.Pr

ince

ton,

NJ:

Prin

ceto

n U

nive

rsity

Pre

ss;1

944


123Tim

e tr

ade-

off

(T

TO

)Pr

oper

ties

: giv

en t

hat

th

e T

TO

tec

hn

ique

invo

lves

tra

din

g, it

can

be

argu

ed t

hat

it is

roo

ted

in c

onsu

mer

th

eory

. As

wit

h t

he

SG, t

he

TT

O m

eth

odin

her

entl

y pr

ovid

es t

he

resp

onde

nt

wit

h a

con

stra

ined

ch

oice

th

at e

licit

s ut

ility

num

bers

th

at a

re h

eld

to r

epre

sen

t st

ren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

253

Braz

ier

et a

l.,19

99T

TOG

iven

tha

t SG

Dol

an e

t al

.,362

in

1 w

eek

or le

ss:0

.87*

Whi

lst T

TO is

not

dir

ectly

rel

ated

T

his

tech

niqu

e ha

s pr

oved

to

be a

pra

ctic

alqu

estio

ns a

rea

larg

e ge

nera

l4

wee

ks:0

.81† ;0

.63‡

to a

ny s

peci

fic t

heor

y,its

sac

rific

eR

esul

ts o

f a s

yste

mat

ic r

evie

w

and

acce

ptab

le m

etho

d of

hea

lth-s

tate

colle

cted

by

popu

latio

n sa

mpl

e 3–

6 w

eeks

:0.5

–0.7

5§el

emen

t lin

ks it

to

cons

umer

of t

he u

se o

f hea

lth s

tatu

s va

luat

ions

in a

wid

e va

riet

y of

stu

dies

,tr

aine

d in

ter-

of 3

395,

foun

d th

e 6

wee

ks:0

.63–

0.80

† ;th

eory

362,

404

mea

sure

s in

eco

nom

ic e

valu

atio

nac

hiev

ing

com

plet

ion

rate

s be

twee

n 90

%

view

ers,

the

TTO

met

hod

to

0.85

#

and

100%

58,3

45,3

63,4

63,4

84–4

88,4

91te

chni

que

will

be

con

sist

ent

10 w

eeks

:0.7

3¶Bu

ckin

gham

et

al.62

5al

ign

the

TTO

be r

elat

ivel

y6–

16 w

eeks

:0.8

3**m

etho

d w

ith t

he w

elfa

re e

cono

mic

Arg

ued

to b

e ea

sier

tha

n SG

62ex

pens

ive35

2,40

0La

upac

is e

t al

.624

foun

d (p

rops

);0.

55**

ap

proa

ch o

f ‘co

mpe

nsat

ing

vari

atio

n’,

sim

ilar

resu

lts(n

o pr

ops)

whe

re w

elfa

re g

ain

is m

easu

red

by1

year

:0.6

2††co

mpe

nsat

ing

loss

of s

omet

hing

A

shby

et

al.36

3co

m-

else

tha

t is

val

uabl

e so

tha

t th

epa

red

TTO

sco

res

Cor

rela

tions

und

er-

resp

onde

nt is

ret

urne

d to

the

with

the

ran

k or

deri

ng

take

n w

here

spe

cifie

d:or

igin

al le

vel o

f wel

fare

of s

tate

s gi

ven

by

intr

acla

ss c

oeffi

cien

t**;

resp

onde

nts.

The

Pe

arso

n co

rrel

atio

nIn

terv

al p

rope

rtie

s of

tec

hniq

uere

sults

sho

w a

con

-co

effic

ient

‡,††

;oth

ers

chal

leng

ed,a

s ha

s th

e re

alis

m o

f si

dera

ble

degr

ee o

f un

spec

ified

the

choi

ces

pose

d62

cons

iste

ncy

in r

anki

ng

and

that

ran

k or

deri

ng

TTO

cha

lleng

ed d

ue t

o th

e ce

rtai

nty

was

con

sist

ently

co

ntex

t of

que

stio

ns p

osed

626

refle

cted

in t

he m

ean

TTO

val

ues

Empi

rica

l evi

denc

e su

ppor

ting

time

pref

eren

ce e

ffect

s502,

627

and

the

viol

atio

n of

the

ass

umpt

ion

ofpr

opor

tiona

l TTO

362,

497–

499

chal

leng

e th

e va

lidity

of T

TO

Empi

rica

l stu

dies

hig

hlig

ht t

he

unw

illin

gnes

s of

indi

vidu

als

to t

rade

lif

e ex

pect

ancy

362,

484,

490,

494,

501

* O

’Con

nor41

8 ;† Ch

urch

ill et

al.49

1 ;‡ G

abrie

let

al.42

0 ;§ As

hby

et a

l.363 ;#

Mol

zahn

et

al.49

2 ;¶ D

olan

et

al.36

2 ;** D

olan

et

al.46

2 ;†† To

rran

ce40

0

Appendix 2

124 Pers

on

trad

e-o

ff (

PT

O)

Prop

erti

es: a

lth

ough

th

e te

chn

ique

is s

een

as

intu

itiv

ely

appe

alin

g,25

1it

has

bee

n c

riti

cise

d fo

r a

lack

of

any

theo

reti

cal b

asis

,250,

253,

503

oth

er t

han

psy

chom

etri

cqu

alit

ies

surr

oun

din

g ad

just

men

t or

equ

ival

ent

stim

uli.24

8

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

253

Braz

ier

et a

l.,19

99A

ccep

tabi

lity

of P

TO c

urre

ntly

unk

now

n.U

bel e

t al

.364

note

Patr

ick

et a

l..248

repo

rtA

lthou

gh t

he t

echn

ique

is in

tuiti

vely

How

ever

,giv

en lo

w r

espo

nse

rate

s an

d na

ture

prob

lem

s w

itha

com

para

tivel

y lo

wap

peal

ing,

251

the

theo

retic

al b

asis

isR

esul

ts o

f a s

yste

mat

ic r

evie

wof

que

stio

ns p

osed

,dat

a co

llect

ion

will

co

nsis

tenc

y of

corr

elat

ion

(Pea

rson

;st

ill t

o be

dev

elop

edof

the

use

of h

ealth

sta

tus

requ

ire

inte

rvie

ws24

8,25

1,36

4,50

4re

spon

ses.

11/5

3r

= 0

.60)

with

res

pect

mea

sure

s in

eco

nom

ic

peop

le e

xclu

ded

from

to in

ter-

rate

r re

liabi

lity,

The

tec

hniq

ue h

as b

een

argu

ed t

oev

alua

tion

Res

pond

ents

may

find

que

stio

ns d

iffic

ult

give

n st

udy

due

to fa

iling

to

com

pare

d w

ith t

hepo

sses

s in

terv

al p

rope

rtie

s57,6

1an

dth

at t

hey

have

to

trad

e liv

es.N

ord

foun

d th

at

answ

er t

he r

atio

ning

VAS

(r=

0.7

5–0.

77)

has

been

sai

d to

ask

the

rig

ht

17 o

ut o

f 53

subj

ects

sho

wed

an

unw

illin

gnes

s sc

enar

io (

PTO

)qu

estio

n (i.

e.tr

ade-

offs

bet

wee

n to

tak

e pa

rt25

1co

nsis

tent

ly,al

thou

gh

Nor

d628

repo

rts

that

peop

le)42

3

this

stu

dy w

as s

elf-

rete

st fi

ndin

gs fr

omad

min

iste

red

20 in

divi

dual

PTO

PT

O m

ay b

e se

nsiti

ve t

o fr

amin

gre

spon

ses

(“so

me

effe

cts

(the

arg

umen

ts m

entio

ned

inw

eeks

aft

er t

he fi

rst

the

ques

tions

,the

sta

rt-p

oint

,the

resp

onse

”) s

how

ed

num

bers

in t

he p

airw

ise

com

pari

son,

a m

ean

diffe

renc

e an

d th

e ch

oice

of d

ecis

ion

of 4

0%co

ntex

t251

The

re is

som

e ev

iden

ce o

f int

erna

l va

lidity

,with

PTO

res

pons

es

cons

iste

ntly

ref

lect

ing

a pr

efer

ence

to

tre

at t

hose

in a

wor

se

stat

e.36

4,42

3

SG a

nd 3

ver

sion

s of

PTO

dir

ectly

or

dere

d th

e he

alth

sta

tes

in li

ne

with

the

exp

ecte

d or

deri

ng.T

he

VAS

did

not42

3

cont

inue

d


125Pers

on

trad

e-o

ff (

PT

O)

cont

d

Prop

erti

es: a

lth

ough

th

e te

chn

ique

is s

een

as

intu

itiv

ely

appe

alin

g,25

1it

has

bee

n c

riti

cise

d fo

r a

lack

of

any

theo

reti

cal b

asis

,250,

253,

503

oth

er t

han

psy

chom

etri

cqu

alit

ies

surr

oun

din

g ad

just

men

t or

equ

ival

ent

stim

uli.24

8

Add

itio

nal a

rtic

les

conc

erni

ng h

ealt

h st

atus

mea

sure

s no

t re

view

ed in

Bra

zier

et

al.25

3*

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

464

Hal

l et

al.,

1992

In c

ompa

ring

SG

,TTO

and

VA,S

G w

as fo

und

to b

e th

e m

ost

diffi

cult

tech

niqu

eA

cos

t–ut

ility

ana

lysi

s of

m

amm

ogra

phy

scre

enin

g

416

Shie

ll et

al.,

1993

Prob

lem

s w

ere

note

d w

ith b

oth

the

TTO

23

(30

%)

patie

nts

gave

sim

ilar

and

VAS

resp

onse

s to

the

2 v

alua

tion

The

con

sist

ency

of V

AS

and

met

hods

TTO

tec

hniq

ues

for

elic

iting

pr

efer

ence

sFo

r th

e m

ajor

ity o

f pat

ient

s (4

5;59

%),

the

2 re

sulte

d in

sub

stan

tially

di

ffere

nt v

alue

s.O

f the

se,7

(9%

) pa

tient

s se

lect

ed t

heir

VAS

resp

onse

in

pre

fere

nce

to t

he T

TO a

nd 8

(1

1%)

patie

nts

sele

cted

the

ir T

TO

resp

onse

in p

refe

renc

e to

the

VA

S.T

he r

emai

ning

30

patie

nts

stuc

k w

ith t

heir

ori

gina

l res

pons

es d

espi

te

the

subs

tant

ial d

iffer

ence

in t

he

resp

onse

s gi

ven

and

the

pres

sure

to

be

cons

iste

nt

365

Badi

a et

al.,

1999

294

(98%

) qu

estio

nnai

res

used

in t

he

% o

f res

pond

ents

with

final

ana

lysi

slo

gica

l inc

onsi

sten

cies

:To

ass

ess

inco

nsis

tent

26

% fo

r VA

S;59

% fo

rre

spon

ses

in p

refe

renc

e-T

TO.M

ean

rate

:1.1

4el

icita

tion

met

hods

for

for V

AS;

3.95

for T

TO.

heal

th s

tate

sIn

crea

sing

age

and

lo

wer

leve

ls o

f ed

ucat

ion

posi

tivel

y as

soci

ated

with

in

cons

iste

nt r

espo

nses

fo

r bo

th V

AS

and

TTO

cont

inue

d

Appendix 2

126 Pers

on

trad

e-o

ff (

PT

O)

cont

d

Prop

erti

es: a

lth

ough

th

e te

chn

ique

is s

een

as

intu

itiv

ely

appe

alin

g,25

1it

has

bee

n c

riti

cise

d fo

r a

lack

of

any

theo

reti

cal b

asis

,250,

253,

503

oth

er t

han

psy

chom

etri

cqu

alit

ies

surr

oun

din

g ad

just

men

t or

equ

ival

ent

stim

uli.24

8

Add

itio

nal a

rtic

les

conc

erni

ng h

ealt

h st

atus

mea

sure

s no

t re

view

ed in

Bra

zier

et

al.25

3* co

ntd

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

494

Rob

inso

n et

al.,

1997

29/4

3 re

spon

dent

s ra

nked

and

sco

red

at le

ast

1 st

ate

abov

e de

ath

on t

he V

AS

but

subs

eque

ntly

rat

ed t

hat

stat

eVa

luin

g he

alth

sat

es u

sing

as

wor

se t

han

dead

in t

he T

TO.W

hen

aske

d to

exp

lain

VAS

and

TTO

thei

r de

cisi

on,a

ll 29

con

firm

ed t

hat

they

wou

ld r

athe

r di

eim

med

iate

ly t

han

spen

d 10

yea

rs in

the

giv

en h

ealth

sta

te.

Whe

n as

ked

whe

ther

the

ir V

AS

answ

er m

eant

tha

t th

eype

rson

ally

pre

ferr

ed 1

0 ye

ars

in t

hat

heal

th s

tate

to

imm

edia

te d

eath

,12/

29 s

aid

that

it d

id,a

nd 1

4 sa

id t

hat

it di

d no

t,w

hils

t 3

did

not

know

15/4

3 re

spon

dent

s (3

4.8%

) re

fuse

d to

tra

de-o

ff ev

ena

few

day

s or

wee

ks in

ord

er t

o av

oid

a he

alth

sta

te t

hat

they

had

pla

ced

belo

w 1

1111

on

the

VAS.

All

15 c

onfir

med

that

the

ir V

AS

resp

onse

mea

nt t

hat

they

con

side

red

10 y

ears

in t

hat

stat

e to

be

wor

se t

han

10 y

ears

in11

111.

The

y di

d no

t,ho

wev

er,s

eem

to

tran

slat

e th

isin

to a

will

ingn

ess

to t

rade

-off

time

to a

void

tha

t st

ate.

Onl

y 1/

15 d

id n

ot t

rade

-off

any

time

at a

ll th

roug

hout

the

TTO

exe

rcis

e,ap

pare

ntly

obj

ectin

g to

the

tas

k on

relig

ious

gro

unds

.Thu

s,th

e au

thor

s st

ate

that

it d

oes

not

appe

ar t

hat

refu

sing

to

‘pla

y th

e ga

me’

in t

he T

TOw

ould

,in

itsel

f,ac

coun

t fo

r th

is d

ispa

rity

.Rat

her,

the

pred

omin

ant

mes

sage

was

tha

t,as

long

as

they

cou

ldco

pe w

ith t

he s

tate

in q

uest

ion,

they

wou

ld n

ot c

on-

side

r gi

ving

up

any

of t

he 1

0 ye

ars

to a

void

it

The

aut

hors

cite

tha

t TTO

sta

tes

had

a ‘th

resh

old

tole

rabi

lity’

bel

ow w

hich

sta

tes

have

to

fall

for

resp

onde

nts

to b

e w

illin

g to

tra

de a

ny t

ime

at a

ll.T

hus,

TTO

may

not

be s

uita

ble

for

use

in c

erta

in c

linic

al s

ettin

gs

Youn

ger

resp

onde

nts

(in T

TO)

find

the

wor

se t

han

dead

sce

nari

o le

ss p

laus

ible

tha

n ol

der

resp

onde

nts

* O

ther

stu

dies

iden

tifie

d,no

t re

view

ed in

Bra

zier

et

al.,

are

men

tione

d in

cho

ice-b

ased

CA19

2,62

0an

d W

TP41

7,45

6,47

0,49

6

cont

inue

d


127Pers

on

trad

e-o

ff (

PT

O)

cont

d

Prop

erti

es: a

lth

ough

th

e te

chn

ique

is s

een

as

intu

itiv

ely

appe

alin

g,25

1it

has

bee

n c

riti

cise

d fo

r a

lack

of

any

theo

reti

cal b

asis

,250,

253,

503

oth

er t

han

psy

chom

etri

cqu

alit

ies

surr

oun

din

g ad

just

men

t or

equ

ival

ent

stim

uli.24

8

Add

itio

nal a

rtic

les

conc

erni

ng h

ealt

h st

atus

mea

sure

s no

t re

view

ed in

Bra

zier

et

al.25

3* co

ntd

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

458

Shac

kley

& C

airn

s,19

9652

elig

ible

wom

en w

ere

inte

rvie

wed

:3 w

ere

Evid

ence

of i

nter

nal v

alid

ity w

ithex

clud

ed fr

om a

naly

sis

beca

use

they

wou

ld

41/4

9 (8

4%)

resp

onde

rs w

illin

g to

Ev

alua

ting

the

bene

fits

of

not

have

a d

iagn

ostic

tes

t un

der

any

acce

pt a

n in

crea

se in

the

ris

k of

feta

l an

tena

tal s

cree

ning

usi

ng S

Gci

rcum

stan

ces

loss

as

thei

r ri

sk o

f cys

tic fi

bros

is

incr

ease

s

* O

ther

stu

dies

iden

tifie

d,no

t re

view

ed in

Bra

zier

et

al.,

are

men

tione

d in

cho

ice-b

ased

CA19

2,62

0an

d W

TP41

7,45

6,47

0,49

6

Appendix 2

128 Will

ingn

ess

to p

ay (

WT

P)

Prop

erti

es: h

as it

s th

eore

tica

l bas

is in

wel

fare

eco

nom

ic t

heo

ry50

5an

d is

hel

d to

mea

sure

str

engt

h o

f pr

efer

ence

.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

257

Klo

se,1

999

Evid

ence

of n

on-r

espo

nse

and

prot

est

Rep

rodu

cibi

lity

Evid

ence

of i

nter

nal v

alid

ity t

hrou

gh p

ositi

vean

swer

s w

ith t

he O

E m

etho

dra

rely

inve

sti-

influ

ence

of i

ncom

e (a

nd in

sev

eral

cas

esR

evie

w o

f met

hodo

logi

cal

gate

d419,

508,

519,

527

soci

al c

lass

) an

d he

alth

gai

n in

man

y as

pect

s in

the

use

of W

TP

and

still

to

beW

TP

stud

ies

in h

ealth

care

prov

en51

6

Evid

ence

of c

onve

rgen

t va

lidity

thr

ough

co

rrel

atio

n w

ith o

ther

mea

sure

s of

he

alth

ben

efits

foun

d in

som

e bu

t no

t al

l stu

dies

419,

495,

520

Evid

ence

of s

trat

egic

beh

avio

ur53

2,54

4

Evid

ence

of ‘

yea-

sayi

ng’ w

ith C

E m

etho

d,35

6

rang

e bi

as w

ith P

C35

5,50

6an

d st

artin

g po

int

bias

with

bid

ding

gam

e353,

356,

552,

558

532

And

erso

n et

al.,

1997

464

resp

onse

s to

the

WT

P qu

estio

n,by

So

me

evid

ence

of t

heor

etic

al v

alid

ity,t

houg

hte

leph

one;

no d

ata

on r

espo

nse

rate

inco

me

was

not

sta

tistic

ally

sig

nific

ant

Empi

rica

l stu

dy o

n ca

tara

ct

surg

ery

wai

ting

list

patie

nts’

W

TP

com

pare

d w

ith a

ctua

l beh

avio

urW

TP

for

shor

ter

wai

ting

times

(sin

ce r

espo

nden

ts w

ere

able

to

have

the

surg

ery

at p

riva

te c

linic

s).F

rom

the

CE

met

hod

3 ce

ntre

s,25

%,1

5% a

nd 1

2% s

aid

they

w

ere

will

ing

to p

ay t

he m

arke

t pr

ice

to

redu

ce w

aitin

g by

1 m

onth

;how

ever

,on

ly 1

.7%

act

ually

did

so

530

App

el e

t al

.,19

9095

(ou

t of

100

) ou

tpat

ient

s w

ere

able

to

Som

e ev

iden

ce o

f int

erna

l val

idity

,tho

ugh

com

plet

e th

e qu

estio

nnai

reso

me

resu

lts w

ere

oppo

site

to

a pr

iori

WT

P fo

r re

duce

d ri

sk o

f low

ex

pect

atio

nsos

mol

ality

con

tras

t m

edia

.Use

s bi

ddin

g ga

me

appr

oach

No

resp

onde

nts

wer

e w

illin

g to

pay

mor

efo

r a

redu

ced

risk

of t

he m

inor

sid

e-ef

fect

sth

an fo

r a

redu

ced

risk

of b

oth

maj

or a

ndm

inor

sid

e-ef

fect

s.H

owev

er,1

2% w

ere

will

ing

to p

ay m

ore

for

a re

duce

d ri

sk in

1 o

f the

8si

de-e

ffect

s th

an a

red

uctio

n in

all

8,an

d 22

%w

ere

will

ing

to p

ay m

ore

for

a re

duce

d ri

skin

1 o

f the

min

or s

ide-

effe

cts

than

for

are

duce

d ri

sk in

all

4 m

inor

sid

e-ef

fect

s cont

inue

d


129Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

470

Bala

et

al.,

1998

Focu

s gr

oups

rev

eale

d bo

th S

G a

nd W

TP

No

diffe

renc

e R

espo

nden

ts g

ave

mea

ning

ful a

nsw

ers

for

both

ques

tions

to

be s

trai

ghtfo

rwar

din

con

sist

ency

SG a

nd W

TP

Com

pari

son

of W

TP

and

SG

betw

een

WT

Pfo

r va

luin

g he

alth

out

com

es

and

SGN

o si

gnifc

ant

rela

tions

hip

betw

een

SG a

ndas

soci

ated

with

shi

ngle

sW

TP,

ques

tioni

ng w

heth

er t

he 2

app

roac

hes

are

equi

vale

nt p

refe

renc

e-ba

sed

outc

ome

Dou

ble-

boun

ded

CE

met

hod

mea

sure

s

549

Baro

n,19

9795

(fr

om 1

08)

stud

ents

com

plet

ed t

he

Expe

rimen

t 1

first

que

stio

nnai

re;2

9 co

mpl

eted

the

sec

ond

WT

P un

resp

onsi

ve t

o qu

antit

y of

live

s sa

ved.

WT

P fo

r re

lativ

e an

d M

ean

WT

P $1

43 fo

r 90

0 liv

es s

aved

and

$74

abso

lute

ris

k re

duct

ions

for

90.C

oncl

uded

tha

t re

spon

dent

s w

ere

influ

ence

d by

pro

port

ion

of li

ves

save

dO

E m

etho

d(m

easu

red

agai

nst

tota

l num

ber

at r

isk)

as

wel

l as

the

num

ber

save

d

Expe

rimen

t 2

Cor

rela

tion

rang

ing

from

0.8

to

1 be

twee

n W

TP

for

a pe

rcen

tage

and

WT

P fo

r nu

mbe

r re

duct

ion

in d

eath

in 2

6% o

f res

pond

ents

.T

his

indi

cate

s re

spon

dent

s in

sens

itive

to

qua

ntity

456

Blum

ensc

hein

&

69 p

atie

nts

com

plet

ed t

he q

uest

ionn

aire

R

atin

g sc

ales

gav

e lo

wes

t ut

ility

mea

sure

and

Jo

hann

esso

n,19

98at

inte

rvie

w.N

o on

e re

fuse

d to

ans

wer

SG h

ighe

stqu

estio

ns o

r ga

ve o

bvio

us ‘p

rote

st’ a

nsw

ers

Rel

atio

nshi

p be

twee

n qu

ality

of

No

sign

ifica

nt d

iffer

ence

bet

wee

n bi

ddin

g an

dlif

e te

chni

ques

(ra

ting

scal

es,S

G,

CE

WT

P,al

thou

gh t

he C

E m

ean

was

hig

her

TTO

) an

d W

TP

Use

d bo

th b

iddi

ng a

nd C

E W

TP

appr

oach

es

cont

inue

d

Appendix 2

130 Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

356

Che

stnu

t et

al.,

1996

50 m

en w

ith a

his

tory

of c

hest

pai

n:5

resp

onde

nts

gave

a lo

wer

res

pons

e to

the

2 re

fuse

d C

E va

rian

t an

d 1

refu

sed

OE

OE

ques

tion

than

the

y in

dica

ted

‘yes

’ to

in t

heW

TP

for

chan

ges

in

CE

ques

tion.

Res

ults

for

the

2 m

etho

ds g

ave

angi

na s

ympt

oms

cons

iste

nt a

nsw

ers

– no

t su

rpri

sing

sin

ce t

hesa

me

resp

onde

nts

answ

ered

bot

h qu

estio

nsTr

iple

-bou

nded

CE

ques

tion

follo

wed

by

OE

ques

tion:

Evid

ence

of ‘

yea-

sayi

ng’ i

n C

E re

spon

ses,

resp

onde

nts

com

plet

ed b

oth

star

ting

poin

t bi

as in

OE

WT

P no

t se

nsiti

ve t

o sc

ope

effe

cts

– m

ean

resp

onse

s no

t si

gnifi

cant

ly d

iffer

ent

for

vary

ing

amou

nts

of a

ngin

a pr

even

tion.

Som

e re

spon

dent

s in

dica

ted

they

wer

e co

ncen

-tr

atin

g on

wha

t th

ey c

ould

affo

rd t

o pa

yra

ther

tha

n th

e qu

antit

y of

the

inte

rven

tion

355

Don

alds

on e

t al

.,19

9751

% (

of 4

50)

resp

onse

rat

e.A

utho

rs n

ote

23%

of

Soci

al c

lass

sig

nific

antly

ass

ocia

ted

with

WT

Pth

at g

iven

the

low

res

pons

e ra

te,c

ondu

ctin

g re

spon

ses

for

met

hod

1.T

he o

nly

sign

ifica

nt v

aria

ble

for

WT

P fo

r cy

stic

fibr

osis

a

post

al W

TP

stud

y ou

tsid

e a

tria

l may

in

cons

iste

nt –

met

hod

2 w

as b

eing

a s

ingl

e m

othe

rca

rrie

r sc

reen

ing

be ‘p

robl

emat

ic’

stat

ed a

pref

eren

ce fo

rO

E qu

estio

ns m

ay n

ot fo

rce

resp

onde

nts

toO

E m

etho

d75

% o

f res

pond

ents

foun

d th

e W

TP

met

hod

1,or

give

the

ir m

axim

um W

TP,

but

rath

er t

oqu

estio

ns d

iffic

ult

to a

nsw

erin

dica

ted

no

indi

cate

an

acce

ptab

le a

mou

ntpr

efer

ence

,but

ga

ve a

hig

her

Mea

n W

TP

sim

ilar

to t

hat

of e

arlie

r st

udy.62

9

WT

P fo

r H

owev

er,t

his

may

be

that

est

imat

es o

f NH

Sm

etho

d 2

cost

s w

ere

sim

ilar

rath

er t

han

WT

P (s

ince

in

cons

iste

nt r

espo

nses

wer

e ex

plai

ned

by

cost

-bas

ed r

espo

nses

)

512

Don

alds

on e

t al

.,19

97H

ighe

r re

spon

se a

nd c

ompl

etio

n ra

te w

ithSt

rong

er a

ssoc

iatio

n w

ith a

bilit

y to

pay

(us

ing

PC a

ppro

ach

(65%

for

PC,6

1% fo

r O

E,no

tth

e pr

oxy

of s

ocia

l cla

ss)

with

PC

app

roac

h.C

ompa

ring

OE

and

PC

stat

istic

ally

sig

nific

ant)

.94%

of r

espo

nden

tsM

ean

WT

P pr

ogre

ssiv

ely

fell

as m

oved

from

appr

oach

esan

swer

ed t

he W

TP

ques

tion

in t

he P

C

soci

al c

lass

I to

Vqu

estio

nnai

re,8

4% in

the

OE

ques

tionn

aire

.N

o ze

ro r

espo

nses

in P

C,6

in O

EM

ean

and

med

ian

WT

P hi

gher

in P

C a

ppro

ach,

alth

ough

onl

y st

atis

tical

ly s

igni

fican

t at

0.

1% le

vel

cont

inue

d


131Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

354

Don

alds

on,1

999

Find

ings

indi

cate

the

met

hod

was

feas

ible

,G

ood

test

–ret

est

WT

P in

sens

itive

to

scop

e ef

fect

s an

d w

ith o

nly

1 of

the

6 c

ount

ries

sho

win

g re

liabi

lity

orde

ring

effe

cts

Rep

ort

from

Eur

opea

n >

10%

pro

test

ans

wer

s to

the

WT

P qu

estio

nC

omm

issi

on s

tudy

to

eval

uate

PC

did

not

dem

onst

rate

ran

ge b

ias

whe

nm

etho

d of

WT

P.In

corp

orat

es

addi

tiona

l bid

s w

ere

adde

d22

50 r

espo

nden

ts a

cros

s 6

Euro

pean

cou

ntri

esIn

sura

nce-

base

d qu

estio

ns le

d to

less

zer

o re

spon

ses

and

high

er W

TP

than

com

mun

ity-

base

d qu

estio

ns

CE

met

hod

resu

lts in

sub

stan

tially

hig

her

estim

ates

com

pare

d w

ith t

he P

C m

etho

d

The

incr

emen

tal o

r m

argi

nal a

ppro

ach

give

s m

ore

cons

iste

nt r

espo

nse

with

sta

ted

rank

ings

The

pro

visi

on o

f add

ition

al in

form

atio

n le

d to

hig

her W

TP

valu

es

521

Dra

nits

aris

,199

7R

ando

m s

ampl

e of

50

patie

nts

rece

ivin

gFa

mily

inco

me,

mar

ital s

tatu

s an

d re

side

nce

chem

othe

rapy

.WT

P te

chni

que

sim

ple

tow

ere

all s

igni

fican

tly a

ssoc

iate

d w

ithW

TP

Feas

ibili

ty o

fWT

P as

a m

easu

re

adm

inis

ter

and

easy

for

resp

onde

nts

of s

uppo

rtiv

e ca

ncer

car

eto

und

erst

and

WT

P or

der

cons

iste

nt w

ith r

atin

g sc

ale

rank

orde

r of

adv

erse

effe

cts

OE

appr

oach

630

East

haug

h,19

91R

espo

nden

ts w

ere

20 b

lood

dis

trib

utio

n A

utho

rs fo

und

dim

inis

hing

WT

P w

ithm

anag

ers

and

50 h

ealth

ser

vice

in

crea

sing

mar

gina

l ben

efit.

For

a re

duct

ion

inVa

lue

of r

isk-

free

blo

odad

min

istr

atio

n gr

adua

te s

tude

nts

risk

of 2

uni

ts (

to z

ero

risk

) m

edia

n W

TP

was

$4

for

man

ager

s an

d $3

for

grad

uate

s.Fo

r a

redu

ctio

n fr

om 4

uni

ts t

o 2

units

it w

as $

2 fo

r bo

th.F

or a

red

uctio

n of

6 u

nits

(9

to 3

) WT

P w

as $

5 an

d $4

.A 1

0-un

it re

duct

ion

(20

to 1

0)

reve

aled

WT

P of

$1

for

both

gro

ups

cont

inue

d

Appendix 2

132 Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

519

Flow

ers

et a

l.,19

9752

“yo

ung

and

wel

l-edu

cate

d” v

olun

teer

s pa

idG

ood

test

–ret

est

relia

bilit

y (0

.796

) at

2 w

eeks

$10

to t

ake

part

.All

com

plet

ed q

uest

ionn

aire

:C

ompu

ter-

base

d as

sess

men

t 71

% in

dica

ted

they

und

erst

ood

the

met

hod,

of W

TP

via

bidd

ing-

type

met

hod

79%

tho

ught

it w

as v

ery

clea

r,84

% t

houg

ht

it us

eful

for

mod

erat

ely

diffi

cult

deci

sion

s,an

d 61

% w

ere

com

fort

able

or

very

com

fort

-ab

le u

sing

WT

P to

mak

e he

alth

care

dec

isio

ns

The

stu

dy in

dica

ted

the

viab

ility

of a

utom

ated

qu

estio

nnai

res

on c

ompu

ter

367

Frew

et

al.,

1999

Part

ial a

naly

sis

so fa

r re

veal

ed h

ighe

r re

spon

se20

% o

fN

eith

er P

C n

or O

E W

TP

estim

ates

wer

era

te w

ith P

C (

85%

and

77%

) th

an O

E (7

0%re

spon

dent

ssi

gnifi

cant

ly a

ssoc

iate

d w

ith s

ocia

l cla

ssW

TP

for

2 al

tern

ativ

e te

sts

for

and

66%

),al

thou

gh n

ot s

tatis

tical

ly s

igni

fican

tw

ho p

refe

rred

colo

rect

al c

ance

r sc

reen

ing

the

first

tes

t or

No

stat

istic

al d

iffer

ence

bet

wee

n m

ean

WT

Pex

pres

sed

nofo

r O

E an

d PC

met

hods

OE/

PC m

etho

ds c

ompa

riso

npr

efer

ence

,gav

e a

high

er W

TP

for

the

seco

nd

test

546

Gar

bacz

& T

haye

r,19

83In

sign

ifica

nt d

iffer

ence

bet

wee

n bi

d fo

r a

25%

and

75%

red

uctio

n in

the

ser

vice

Valu

ing

a se

nior

com

pani

on

prog

ram

me

(WT

P an

d W

TA)

Equi

vale

nt v

aria

tion

bids

wer

e ar

ound

hal

f of

the

com

pens

atin

g va

riat

ion

543

Gra

nber

g et

al.,

1995

55%

of c

oupl

es a

nsw

erin

g th

e W

TP

ques

tion

wer

e w

illin

g to

pay

mor

e th

an £

10,0

00 fo

r IV

FVa

lue

of IV

F se

rvic

es u

sing

tr

eatm

ent,

whi

ch e

quat

es w

ith t

rue

cost

of

OE

appr

oach

£941

0

506

Joha

nnes

son

et a

l.,19

91R

espo

nse

rate

was

99%

for

PC1

and

97%

The

diff

eren

ce b

etw

een

the

mea

n of

the

for

PC2

2 PC

met

hods

not

sta

tistic

ally

sig

nific

ant

Cos

t–be

nefit

ana

lysi

s of

non

-ph

arm

acol

ogic

al t

reat

men

t of

hy

pert

ensi

on,i

ncor

pora

ting

WT

P fo

r be

nefit

ass

essm

ent

2 PC

met

hods

cont

inue

d


133Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

263

Joha

nnes

son

et a

l.,19

9364

% r

espo

nse

rate

(33

5/52

5):i

tem

non

-Bi

d an

d he

alth

sta

tus

sign

ifica

ntly

ass

ocia

ted

resp

onse

was

5%

on

the

WT

P qu

estio

nw

ith W

TP

but

none

of t

he s

ocio

-eco

nom

icW

TP

for

antih

yper

inte

nsiv

e co

mpa

red

with

18%

on

the

VAS

vari

able

wer

eth

erap

yPo

ssib

le r

ange

bia

s as

upp

er b

id s

et b

y au

thor

sU

sed

CE

ques

tion

with

5-p

oint

to

whi

ch 2

4% r

espo

nded

in t

he a

ffirm

ativ

eLi

kert

-typ

e sc

ale

inco

rpor

atin

g a

‘don

’t kn

ow’ r

espo

nse

552

Kar

tman

et

al.,

1996

341/

438

(78%

) re

spon

ses:

36 r

efus

ed t

o M

ean

WT

P fo

r bo

th m

etho

ds w

as s

imila

rpa

rtic

ipat

e,41

pro

test

ans

wer

s an

d 20

C

ompa

ring

CE

with

bid

ding

m

issi

ng v

alue

sEv

iden

ce o

f int

erna

l val

idity

,with

mea

n W

TP

gam

e in

the

con

text

of a

ngin

a st

atis

tical

ly s

igni

fican

tly a

nd p

ositi

vely

rel

ated

pect

oral

att

acks

wee

kly

atta

ck r

ates

and

ang

ina

pect

oris

sta

tus

Both

met

hods

indi

cate

WT

P in

crea

sed

with

sc

ope

in t

he c

ase

of s

ize

of r

educ

tion

in a

ngin

a at

tack

s.T

his

was

hig

hly

stat

istic

ally

sig

nific

ant

in t

he b

iddi

ng g

ame

but

not

stat

istic

ally

si

gnifi

cant

in C

E m

etho

d

Star

ting

poin

t bi

as d

etec

ted

in t

he b

iddi

ng

gam

e

553

Kar

tman

et

al.,

1996

400/

461

patie

nts

resp

onde

d (1

3% n

on-

WT

P in

crea

sed

with

the

pro

babi

lity

of

resp

onse

rat

e) v

ia t

elep

hone

inte

rvie

ws

bein

g fr

ee fr

om s

ympt

oms

and

redu

ced

Test

ing

the

CE

WT

P ap

proa

ch

by a

nur

seri

sk o

f rel

apse

for

scop

e an

d qu

estio

n or

der

effe

cts

Met

hod

sens

itive

to

chan

ges

in s

cope

No

ques

tion

orde

r ef

fect

s w

ere

foun

d

509

Kar

tman

et

al.,

1997

Zer

o re

spon

se fo

r th

e O

E qu

estio

n ra

nged

Met

hod

inse

nsiti

ve t

o sc

ope

effe

cts

from

7%

to

25%

and

non

-res

pons

e ra

nged

Test

ing

the

CE

met

hod

with

fr

om 7

% t

o 13

%O

E fo

llow

-up

met

hod

for

scop

e ef

fect

s

cont

inue

d

Appendix 2

134 Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

547

Kob

elt,

1997

85%

res

pons

e ra

te (

excl

udin

g th

ose

who

had

R

espo

nden

ts w

illin

g to

pay

alm

ost

twic

e di

fficu

lties

in a

nsw

erin

g th

e W

TP

ques

tion

led

as m

uch

for

doub

le t

he im

prov

emen

t in

WT

P fo

r a

redu

ctio

n in

to

red

uctio

n in

usa

ble

resp

onse

s to

66%

)sy

mpt

oms.

Prop

ortio

n of

pat

ient

s w

illin

g in

cont

inen

ce s

ympt

oms

to p

ay a

lso

incr

ease

d fo

r th

e hi

gher

leve

l of

ben

efit

CE

met

hod

WT

P po

sitiv

ely

and

sign

ifica

ntly

ass

ocia

ted

with

sev

erity

of s

ympt

oms

417

Kra

bbe

et a

l.,19

97U

sing

dom

inan

tG

ood

agre

emen

t be

twee

n ra

nkin

g of

hea

lthpa

irs,

inco

n-st

ates

and

val

uatio

ns:r

atin

g =

0.8

3,SG

= 0

.75,

Com

para

bilit

y an

d re

liabi

lity

of

sist

enci

es fo

rT

TO =

0.7

7,W

TP

= 0

.80

SG,T

TO,W

TP

and

ratin

g sc

ales

WT

P va

ried

from

1.9

% t

oG

ood

conv

erge

nt v

alid

ity b

etw

een

SG a

ndO

E m

etho

d50

.5%

dep

endi

ng

TTO

,but

not

so

good

for

othe

r co

mpa

risi

ons

on d

ista

nce

betw

een

rate

s

For

ratin

g sc

ales

,ra

nge

= 0

–12.

5,SG

= 0

–21.

6,T

TO =

1.9

–17.

8

WT

P C

ronb

ach’

s α

= 0

.77

Rat

ing

Cro

nbac

h’s

α=

0.5

8SG

Cro

nbac

h’s

α=

0.6

5T

TO C

ronb

ach’

s α

= 0

.49

522

Lee

et a

l.,19

97R

espo

nden

ts e

nter

ed in

to lo

tter

y to

win

WT

P in

crea

sed

sign

ifica

ntly

with

inco

me,

$50

gift

tok

en.4

4% r

espo

nse

rate

(23

5/52

8):

drea

d of

allo

geni

c tr

ansf

usio

n,an

d pe

rcei

ved

Patie

nts’

WT

P fo

r au

tolo

gous

5.

5% d

id n

ot a

nsw

er t

he W

TP

ques

tion,

94%

risk

of r

equi

ring

a b

lood

tra

nsfu

sion

bloo

d do

natio

nra

ted

the

ques

tionn

aire

‘eas

y or

ver

y ea

sy’

CE

met

hod

cont

inue

d


135Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

518

Lind

holm

et

al.,

1997

Res

pons

e ra

tes

96%

(10

0/10

4) fo

r WT

PEv

iden

ce c

once

rnin

g in

tern

al v

alid

ity m

ixed

,qu

estio

n w

ith 5

% r

educ

tion

in r

isk,

and

94%

with

som

e va

riab

les

mov

ing

in t

he e

xpec

ted

WT

P fo

r pr

even

tion

of

(98/

104)

for

the

10%

red

uctio

n.3

outli

ers

dire

ctio

n,an

d ot

her

mov

ing

oppo

site

to

card

iova

scul

ar d

isea

setr

eate

d as

pro

test

s ba

sed

on in

terv

iew

ers

a pr

iori

hypo

thes

esno

tes

Bidd

ing

gam

e

527

Loeh

man

et

al.,

1979

22.4

% (

404/

1800

) re

spon

se r

ate

Test

ed o

n a

sepa

rate

sam

ple

of 4

7 co

llege

stud

ents

and

re-

adm

inis

tere

d af

ter

3 w

eeks

Cos

ts a

nd b

enef

its o

f air

fo

und

aver

age

corr

elat

ion

of 0

.85

(ran

gequ

ality

con

trol

in a

n ur

ban

0.82

–0.9

5)re

gion

in F

lori

da u

sing

WT

P to

val

ue h

ealth

effe

cts

PC a

ppro

ach

368

Mie

dzyb

rodz

ka e

t al

.,19

95O

f 450

res

pons

es,1

73 (

38%

) ga

veW

TP

for

the

step

wis

e m

etho

d w

asin

cons

iste

nt r

espo

nses

(i.e

.wer

e w

illin

gsi

gnifi

cant

ly a

ssoc

iate

d w

ith s

ocia

l cla

ss.B

eing

Wom

en’s

pref

eren

ces

for

to p

ay le

ss fo

r th

eir

pref

erre

d op

tion)

sing

le (

not

mar

ried

or

livin

g w

ith s

omeo

ne)

cyst

ic fi

bros

is s

cree

ning

met

hod

was

sig

nific

antly

ass

ocia

ted

with

WT

P fo

r th

eco

uple

met

hod;

soci

al c

lass

was

not

.Edu

-O

E m

etho

dca

tion

was

sig

nific

antly

ass

ocia

ted

with

nei

ther

631

Mill

s et

al.,

1994

Stat

ed W

TP

low

er t

han

cost

of i

mpr

egna

ting

nets

exc

ept

in 1

reg

ion.

Res

pond

ents

like

ly t

oFu

nd r

aisi

ng fo

r vi

llage

un

ders

tate

WT

P in

the

hop

e of

sub

sidi

esac

tiviti

es in

The

Gam

bia

548

Mul

ler

& R

eutz

el,1

984

89%

res

pons

e ra

te (

77/8

7)Ev

iden

ce fo

und

of s

cope

effe

cts.

Res

pond

ents

’W

TP

for

twic

e th

e ri

sk r

educ

tion

was

muc

hW

TP

for

a re

duct

ion

in

grea

ter

fata

lity

risk

thr

ough

car

cr

ash

prot

ectio

nA

com

pari

son

of m

edia

ns s

ugge

sted

a h

ighe

r W

TP

whe

n th

e ri

sk w

as e

xpre

ssed

out

of

10,0

00 r

athe

r th

an 1

00,a

lthou

gh t

he r

isk

was

the

sam

e

Res

pond

ents

’ WT

P w

as a

bout

1/3

hig

her

whe

n as

ked

to p

ay a

mon

thly

am

ount

rat

her

than

an

annu

al s

um

cont

inue

d

Appendix 2

136 Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

419

O’B

rien

& V

iram

onte

s,19

9477

% r

espo

nse

rate

(10

2/13

3 in

terv

iew

s:4-

wee

k te

st–r

etes

tW

TP

sign

ifica

ntly

ass

ocia

ted

with

hou

seho

ld

16 c

ould

not

be

cont

acte

d,13

ref

used

to

relia

bilit

y de

emed

inco

me

Com

pari

ng W

TP

(bid

ding

gam

e)

part

icip

ate

and

3 ha

d la

ngua

ge o

rac

cept

able

at

0.66

with

SG

,sho

rt-fo

rm 3

6 (S

F-36

)

hear

ing

prob

lem

s)(lo

wer

tha

n SG

No

evid

ence

of s

tart

ing

poin

t bi

asan

d a

ratin

g sc

ale

= 0

.82)

SG w

as m

ost

high

ly r

elat

ed t

o W

TP

(r =

0.4

6)

com

pare

d w

ith a

rat

ing

scal

e an

d SF

-36

534

O’B

rien

et

al.,

1998

501

peop

le r

eplie

d th

ey w

ould

be

inte

rest

edW

TP

sign

ifica

ntly

ass

ocia

ted

with

the

siz

e of

out

of 3

486

lett

ers

sent

out

.220

inte

rvie

wed

.ri

sk r

educ

tion,

alth

ough

at

a de

clin

ing

mar

gina

lFe

asib

ility

and

val

idity

of W

TP

$25

ince

ntiv

e pa

ymen

tra

te.W

TP

high

er in

the

low

er p

rior

ris

k gr

oup

for

new

dru

g fo

r fe

brile

ne

utro

peni

aW

TA e

xcee

ded

WT

P.2

algo

rith

ms

with

the

bidd

ing

gam

e w

ere

used

to

test

for

bias

.Bi

ddin

g ga

me

appr

oach

,A

lthou

gh t

he s

econ

d sc

enar

io g

ave

a hi

gher

2 al

gori

thm

sm

ean

resp

onse

,it

was

not

sta

tistic

ally

si

gnifi

cant

No

star

ting

poin

t bi

as d

etec

ted

262

Ols

en &

Don

alds

on,1

998

Subj

ects

inte

rvie

wed

in t

heir

ow

n ho

mes

by

Dis

crep

anci

esN

o ev

iden

ce t

o su

ppor

t hy

poth

esis

tha

t m

entr

aine

d in

terv

iew

ers.

64%

res

pons

e ra

tebe

twee

n W

TP

and

wou

ld h

ave

high

er W

TP

valu

es fo

r he

art

Usi

ng W

TP

to s

et p

rior

ities

(1

50/2

35):

7 ex

clud

ed a

s w

ould

nei

ther

ordi

nal r

anki

ng.

oper

atio

ns,a

lthou

gh w

omen

had

a h

ighe

r WT

Pbe

twee

n 3

alte

rnat

ive

expr

ess W

TP

nor

prio

ritis

e al

tern

ativ

eFo

r 3

alte

rnat

ive

for

hip

oper

atio

ns.E

duca

tion

had

a si

gnifi

cant

prog

ram

mes

prog

ram

mes

prog

ram

mes

,ne

gativ

e as

soci

atio

n w

ith t

he h

elic

opte

ral

thou

gh o

nly

25%

ambu

lanc

e pr

ogra

mm

e bu

t w

as n

ot a

ssoc

iate

dPC

met

hod

(35)

gav

e st

rong

with

eith

er o

f the

2 o

ther

pro

gram

mes

orde

ring

bet

wee

n al

l 3 in

ter

ms

of

Som

e ev

iden

ce t

hat

leng

th o

f int

ervi

ew t

ime

WT

P,76

% (

109)

did

w

as p

ositi

vely

ass

ocia

ted

with

WT

Pin

ter

ms

of t

heir

or

dina

l ran

king

.Of

WT

P va

lues

for

the

helic

opte

r pr

ogra

mm

esth

e 40

res

pond

ents

wer

e si

mila

r to

tha

t ch

arge

d fo

r th

e ac

tual

who

wer

e in

diffe

rent

prog

ram

me

in e

xist

ence

in t

erm

s of

WT

Pac

ross

the

3 p

ro-

Onl

y 25

% o

f res

pond

ents

had

com

pare

d th

egr

amm

es,2

7 ga

ve

diffe

rent

num

bers

of p

atie

nts

invo

lved

in e

ach

stro

ng p

refe

renc

es

prog

ram

me

and

for

the

othe

r 75

% it

was

the

in t

erm

s of

ord

inal

pr

ogra

mm

es t

hem

selv

es t

hat

wer

e co

nsid

ered

rank

ings

in t

he d

ecis

ion-

mak

ing

proc

ess55

1

cont

inue

d


137Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

528

Penn

ie e

t al

.,19

9195

% o

f stu

dent

s w

ho a

tten

ded

clas

ses

Rel

iabi

lity

was

Stud

ents

con

side

ring

the

mse

lves

at

the

leas

ton

the

que

stio

nnai

re d

ay c

ompl

eted

test

ed w

ith a

risk

(as

aga

inst

hig

her-

risk

med

ical

labo

rato

ryEm

piri

cal s

tudy

inco

rpor

atin

g th

e qu

estio

nnai

rese

para

te g

roup

of

stud

ents

) an

d th

eref

ore

with

the

leas

t to

gai

nst

uden

ts’ W

TP

for

a he

patit

is

nurs

ing

stud

ents

wer

e th

e le

ast

will

ing

to p

ay t

he r

etai

l pri

ceB

vacc

ine

2 w

eeks

late

r an

dth

ere

wer

e no

PC

app

roac

hsi

gnifi

cant

diff

eren

ces

353

Phill

ips

et a

l.,19

97R

espo

nse

rate

s w

ere

71%

(of

396

) pe

ople

Litt

le e

vide

nce

of fr

ee r

idin

g* ,but

som

e w

ho h

ad c

alls

blo

cked

due

to

budg

et c

uts,

evid

ence

of s

tart

ing

poin

t bi

asW

TP

for

pois

on c

ontr

ol

72%

(of

418

) w

ho c

alle

d af

ter

the

bloc

k w

asce

ntre

s in

the

USA

lifte

d,an

d 48

% (

of 1

19)

from

the

gen

eral

popu

latio

nBi

ddin

g ga

me

appr

oach

535

Ram

sey

et a

l.,19

97Po

stal

sur

vey:

85%

res

pons

e ra

te (

from

194

)In

com

e an

d ed

ucat

ion

incr

ease

d as

WT

Pfo

r pa

ymen

t ar

m (

resp

onde

nts

wer

e pa

id $

3in

crea

sed.

Lack

of s

igni

fican

t as

soci

atio

nW

TP

for

anti-

hype

riea

ch,a

lthou

gh 1

0% o

f the

mon

ey w

asbe

twee

n W

TP

and

prio

r hi

stor

y of

str

oke,

nten

sive

car

ere

turn

ed)

and

76%

(of

206

) fr

om t

he n

on-

myo

card

ial i

nfar

ctio

n an

d pe

rcei

ved

paym

ent

arm

.30%

gav

e an

“un

inte

rpre

tabl

e”he

alth

sta

tus

PC m

etho

d,al

thou

gh u

sing

a

answ

er t

o th

e W

TP

ques

tion.

Like

lihoo

d of

5-po

int

Like

rt s

cale

inst

ead

of

unin

terp

reta

ble

answ

er in

crea

sed

with

age

.‘y

es/n

o’ fo

r ea

ch b

id (

yes,

Des

ign

may

hav

e co

nfus

ed b

ecau

se s

ome

defin

itely

;yes

,pro

babl

y;no

,an

swer

ed ‘d

on’t

know

’ to

ever

y bi

dpr

obab

ly;n

o,de

finite

ly;

don’

t kn

ow)

540

Reu

tzel

& F

urm

aga,

1993

198

patie

nts

wer

e in

terv

iew

ed.M

ean

age

47/1

46 g

ave

sam

e W

TP

usin

g bo

th p

aym

ent

63 y

ears

.146

pro

vide

d re

spon

ses

to b

oth

vehi

cles

.The

2 m

etho

ds d

id n

ot g

ive

the

sam

eW

TP

for

phar

mac

y se

rvic

es

WT

P qu

estio

nsan

swer

s,bu

t re

spon

ses

not

stat

istic

ally

in a

Vet

eran

’s ho

spita

l in

sign

ifica

ntly

diff

eren

tth

e U

SAW

hils

t so

me

evid

ence

of i

nter

nal v

alid

ity,

2 pa

ymen

t ve

hicl

es u

sed:

vari

ance

in W

TP

was

une

xpla

ined

.Thi

s m

aypa

rtic

ipat

ion

cost

s (t

rave

l in

dica

te t

hat

resp

onde

nts

‘gue

ss’ W

TP

valu

esco

sts

etc)

and

a fe

e

*Fr

ee r

idin

g re

fers

to

the

tend

ency

to

unde

rest

imat

e or

ove

rest

imat

e tr

ue W

TP,d

epen

ding

on

the

ince

ntive

s in

here

nt in

the

que

stio

n

cont

inue

d

Appendix 2

138 Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

260

Rya

n et

al.,

1997

704

of 9

56 (

74%

) re

turn

ed t

he p

osta

lIn

dire

ct e

vide

nce

was

foun

d fo

r th

equ

estio

nnai

re.6

58 (

93%

) of

the

se c

ompl

eted

inse

nsiti

vity

of W

TP

to t

he t

ype

of c

are

Valu

ing

alte

rnat

ive

form

s th

e W

TP

sect

ion

of a

nten

atal

car

e

CE

appr

oach

559

Rya

n et

al.,

1998

CE

WT

P se

nsiti

ve t

o tr

eatm

ent

of ‘d

on’t

know

’re

spon

ses

but

not

non-

dem

ande

rs o

r le

vel

Com

pari

son

of P

C a

nd

of in

tegr

atio

nC

E ap

proa

ches

555

Rya

n et

al.,

1999

The

CE

appr

oach

gav

e ri

se t

o si

gnifi

cant

lyhi

gher

WT

P es

timat

es;m

ay b

e du

e to

Com

pari

ng C

E an

d w

ith

‘yea

-say

ing’

PC m

etho

ds

560

Rya

n &

Rat

cliff

e,20

0093

% (

658/

704)

of r

espo

nden

ts c

ompl

eted

WT

P se

nsiti

ve t

o th

e lim

its o

f int

egra

tion,

the

WT

P qu

estio

nsbi

d ve

ctor

des

ign

and

met

hod

of a

naly

sis

Issu

es r

aise

d in

ana

lysi

ng C

E W

TP

data

in c

onte

xt o

f an

tena

tal c

are

369

Rya

n &

San

Mig

uel,

2000

51%

res

pons

e ra

te (

75/1

46)

41%

(20

) of

pa

tient

s pr

efer

red

Con

sist

ency

of W

TP

estim

ates

co

nser

vativ

e su

rger

yin

con

text

of 2

alte

rnat

ive

but

wer

e pr

epar

edsu

rgic

al p

roce

dure

s fo

r to

pay

mor

e fo

rm

enor

rhag

iahy

ster

ecto

my.

60%

of r

espo

nden

ts w

hoPC

met

hod

gave

inco

nsis

tent

an

swer

s m

entio

ned

cost

as

the

reas

on

for

thei

r WT

P

cont

inue

d


139Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

496

Swan

et

al.,

1999

29 o

f the

30

resp

onde

nts

answ

ered

the

The

re w

ere

no s

tatis

tical

ly s

igni

fican

tW

TP

ques

tions

.28/

30 a

nsw

ered

the

diffe

renc

es b

etw

een

ratin

g sc

ale,

WT

PPa

tient

s’ u

tility

from

mag

netic

T

TO q

uest

ions

and

TTO

find

ings

reso

nanc

e an

d co

nven

tiona

l an

giog

raph

y

WT

P,T

TO,r

atin

g sc

ales

and

othe

r es

timat

es c

olle

cted

.O

E (%

of i

ncom

e) a

ppro

ach

508

Tho

mps

on e

t al

.,19

84R

espo

nse

rate

27%

(49

of 1

84),

alth

ough

Test

–ret

est

WT

P w

as p

ositi

vely

ass

ocia

ted

with

num

ber

resp

onde

nts

not

pres

sed

for

answ

er.

relia

bilit

y po

or:

of s

ympt

oms.

WT

P w

as “

rela

tivel

y” in

sens

itive

Empi

rica

l stu

dy o

n fe

asib

ility

of

Res

pond

ents

with

mor

e ed

ucat

ion,

in p

aid

corr

elat

ion

to in

com

eW

TP

for

chro

nic

arth

ritis

empl

oym

ent

or w

ho w

ere

havi

ng m

ore

trea

t-co

effic

ient

= 0

.25,

men

t fo

r ar

thri

tis m

ore

likel

y to

res

pond

stat

istic

al s

igni

fican

ceO

E ap

proa

ch(r

ose

to 7

1% fo

r th

ose

in p

aid

empl

oym

ent)

= 0

.08

544

Wal

rave

n,19

9650

0 ou

tpat

ient

s,32

1 in

patie

nts

and

A m

ajor

ity o

f pat

ient

s ac

tual

ly p

aid

mor

e th

an15

00 h

ouse

hold

ers

wer

e in

terv

iew

ed.I

nth

ey s

aid

they

wer

e w

illin

g to

pay

.62%

at

Empi

rica

l stu

dy o

f WT

P fo

r ad

ditio

n,22

focu

s gr

oups

wer

e co

nduc

ted

1 ho

spita

l and

67%

at

anot

her.

At

anot

her

dist

rict

hos

pita

l ser

vice

s ho

spita

l,w

hich

late

r in

trod

uced

use

r ch

arge

s,in

Tan

zani

aW

TP

pred

icte

d be

havi

our

“rea

sona

bly

wel

l”

OE

appr

oach

632

Zei

dner

& S

hech

ter,

1994

Stud

ents

with

“hi

gher

sta

kes”

(i.e

.mor

e to

lose

by b

eing

in a

cla

ss w

ith h

isto

rica

lly a

hig

her

fail

Stud

ents

’ WT

P to

red

uce

thei

r ra

te)

wer

e m

ore

anxi

ous

and

WT

P m

ore

leve

l of a

nxie

ty d

urin

g ex

ams

com

pare

d to

stu

dent

s w

ith “

low

sta

kes”

.St

uden

ts w

ho a

re m

ore

anxi

ous

duri

ng t

ests

ar

e W

TP

mor

e to

red

uce

thei

r an

xiet

y

Stud

ents

’ WT

P w

as a

ssoc

iate

d w

ith

diss

atis

fact

ion

with

cur

rent

anx

iety

leve

ls,a

nd

stud

ents

wer

e W

TP

sign

ifica

ntly

mor

e fo

r la

rger

tha

n sm

alle

r re

duct

ions

in a

nxie

ty

Stud

ents

’ WT

P w

as s

igni

fican

tly le

ss t

han

they

re

com

men

ded

othe

rs s

houl

d pa

y

cont

inue

d

Appendix 2

140 Will

ingn

ess

to p

ay (

WT

P)

cont

dPr

oper

ties

: has

its

theo

reti

cal b

asis

in w

elfa

re e

con

omic

th

eory

505

and

is h

eld

to m

easu

re s

tren

gth

of

pref

eren

ce.

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

495

Zet

hrae

us e

t al

.,19

97M

ay b

e ra

nge

bias

in a

naly

sis

sinc

e cu

t-of

fup

per

valu

e w

as u

sed.

Wom

en w

ith m

ore

Wom

en’s

qual

ity o

f life

with

se

vere

sym

ptom

s ha

d a

high

er W

TP.

WT

Pho

rmon

e re

plac

emen

t th

erap

y va

lues

wer

e co

nsis

tent

with

rat

ing

scal

e an

dfo

r m

ild a

nd s

ever

e m

enop

ausa

l T

TO m

easu

rem

ents

sym

ptom

s

Als

o in

vest

igat

es W

TP

for

horm

one

repl

acem

ent

ther

apy

trea

tmen

t vi

a C

E m

etho

d

Mea

sure

of

valu

e (M

oV

)Pr

oper

ties

: th

roug

h it

s n

atur

e of

ass

ign

ing

num

eric

al s

core

s to

alt

ern

ativ

e op

tion

s, t

he

tech

niq

ue o

f M

oV in

corp

orat

es a

rel

ativ

e st

ren

gth

of

pref

eren

cem

easu

re. A

con

stra

ined

ch

oice

can

als

o be

app

lied

if t

he

sele

ctio

n o

f al

tern

ativ

es a

re p

rice

d, a

nd

resp

onde

nts

mus

t m

ake

thei

r ch

oice

s w

ith

in a

gi

ven

bud

get.26

5,26

6

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

264

Chu

rchm

an &

Ack

off,

1954

The

nat

ure

of t

he t

echn

ique

may

be

quite

Valid

ity s

houl

d be

enc

oura

ged

by n

atur

e of

the

a le

ngth

y an

d tir

esom

e pr

oces

s if

ther

e ar

ete

chni

que.

Onc

e pr

efer

ence

s ar

e ra

nked

,the

yT

he p

ione

erin

g pa

per

deve

lopi

ng

man

y al

tern

ativ

e op

tions

to

com

pare

are

exam

ined

aga

in t

hrou

gh a

com

pari

son

the

tech

niqu

e an

d ill

ustr

atin

g its

ag

ains

t gr

oups

of a

ltern

ativ

es a

nd a

djus

ted

pote

ntia

l usa

geif

nece

ssar

y

265

Dic

kins

on,1

979

Alth

ough

agr

eein

g th

e in

itial

list

of p

rior

ities

was

diff

icul

t,th

e au

thor

not

ed t

he t

ime

and

Met

hodo

logi

cal p

aper

on

beha

lf ef

fort

invo

lved

in m

anag

emen

t tim

e w

asof

Her

efor

d H

ealth

Dis

tric

t to

ar

ound

15

hour

s,an

d w

as ju

dged

a “

good

dete

rmin

e th

e be

st c

ombi

natio

n bu

y”.T

he a

utho

r fu

rthe

r no

ted

that

som

eone

of p

rior

ities

from

a li

st o

f 11

with

a g

ood

know

ledg

e an

d ex

peri

ence

of

‘pri

ced’

hea

lthca

re g

oods

th

e te

chni

que

wou

ld b

e re

quir

edw

ithin

a fi

xed

budg

etas

adm

inis

trat

or


141Allo

cati

on

of

poin

tsPr

oper

ties

: bec

ause

th

e “b

udge

t” o

f po

ints

(to

ken

s, m

oney

, liv

es)

to a

lloca

te is

fix

ed, r

espo

nde

nts

are

sub

ject

ed t

o a

con

stra

ined

ch

oice

.276,

277,

561

Giv

en t

he

wor

din

g of

th

e qu

esti

on, i

t h

as b

een

arg

ued

to p

osse

ss c

ardi

nal

pro

pert

ies.

277

Ho

invi

lle P

EM

*

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

268

Hoi

nvill

e,19

96In

the

con

text

W

hen

pric

e in

crea

sed,

a lo

wer

qua

ntity

was

of

var

iabl

esch

osen

,and

vice

ver

saM

etho

dolo

gica

l pap

er

surr

ound

ing

jour

neys

illus

trat

ing

the

PEM

to

wor

k,th

e re

test

with

exa

mpl

es2

wee

ks la

ter

was

75

% (

of 1

20)

267

Hoi

nvill

e,19

77D

urin

g ea

rly

deve

lopm

ent

of t

he t

echn

ique

,32

res

pond

ents

16 r

espo

nden

ts a

sked

to

allo

cate

poi

nts

only

3%

of r

espo

nden

ts h

ad d

iffic

ulty

.No

aske

d to

rep

eat

betw

een

5 3-

poin

t sc

ales

,on

2 oc

casi

ons,

Met

hodo

logi

cal p

aper

no

ticea

ble

diffe

renc

es a

cros

s so

cio-

econ

omic

an e

xerc

ise

(usi

ng2–

3 w

eeks

apa

rt.O

n th

e se

cond

occ

asio

n,th

eill

ustr

atin

g th

e us

e of

the

an

d ag

e gr

oups

.Mos

t en

joye

d an

d go

tve

rbal

and

pic

tori

alpr

ice

of 1

sca

le w

as in

crea

sed

and

anot

her

PEM

with

exa

mpl

essa

tisfa

ctio

n fr

om in

terv

iew

scal

es)

afte

r a

decr

ease

d.4

resp

onde

nts

gave

the

sam

e2-

wee

k in

terv

al.

answ

ers

whi

lst

9 re

acte

d po

sitiv

ely

to t

he p

rice

24 g

ave

cons

iste

nt

chan

ge.F

rom

a fu

rthe

r 61

1 re

spon

dent

s in

aan

swer

s w

ith t

heir

co

mpu

ter

stud

y on

ly 6

% a

cted

irra

tiona

lly b

ypi

ctor

ial c

hoic

e an

d bu

ying

mor

e w

hen

the

pric

e w

ent

up a

nd le

ss22

with

the

ir v

erba

l w

hen

it w

ent

dow

n.In

ano

ther

stu

dy,o

f 121

choi

ce.U

sing

ver

bal/

resp

onde

nts,

only

1%

mad

e ill

ogic

al c

hoic

espi

ctor

ial s

cale

,out

of

ano

ther

32

Whe

n a

grou

p of

res

pond

ents

wer

e to

ld t

hey

resp

onde

nts,

inte

r-ha

d to

act

on

beha

lf of

loca

l cou

ncils

com

pare

dvi

ewed

a fe

w w

eeks

to

tho

se a

ctin

g fo

r th

emse

lves

,onl

y 1

out

ofap

art,

22 g

ave

con-

10 v

aria

bles

was

sig

nific

antly

diff

eren

t at

the

si

sten

t an

swer

s.5%

leve

lFi

nally

,of 1

7 ot

her

resp

onde

nts

indi

-ca

ting

thei

r pr

efer

-en

ces

on 5

3-p

oint

sc

ales

,7 g

ave

iden

tical

res

pons

es

and

6 ga

ve o

nly

1 sl

ight

diff

eren

ce

* Th

e H

oinv

ille P

EM is

one

of t

he fi

rst

appl

icatio

ns o

f the

allo

catio

n of

poi

nts

appr

oach

Appendix 2

142 Allo

cati

on

of

poin

ts c

ontd

Prop

erti

es: b

ecau

se t

he

“bud

get”

of

poin

ts (

toke

ns,

mon

ey, l

ives

) to

allo

cate

is f

ixed

, res

pon

den

ts a

re s

ubje

cted

to

a co

nst

rain

ed c

hoi

ce.27

6,27

7,56

1G

iven

th

ew

ordi

ng

of t

he

ques

tion

, it

has

bee

n a

rgue

d to

pos

sess

car

din

al p

rope

rtie

s.27

7

Allo

cati

on

of

poin

ts

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

633

Bate

,199

9R

espo

nse

rate

to

a po

stal

que

stio

nnai

reVa

riat

ion

in r

elat

ive

impo

rtan

ce o

f cri

teri

onw

ithou

t re

min

ders

was

29.

4% (

42/1

43).

acro

ss m

etho

ds.H

owev

er,l

ack

of n

umbe

rsIm

port

ance

of p

ublic

vie

ws

11 w

ere

unsu

cces

sful

ly c

ompl

eted

leav

ing

sugg

est

furt

her

wor

kag

ains

t ot

her

crite

ria

for

31 u

sabl

e re

spon

ses

(21.

7%)

prio

rity

set

ting

Com

pari

son

of c

hoic

e-ba

sed

CA

and

bud

get

pie

(allo

catio

n of

poi

nts)

562

Sriv

asta

va e

t al

.,19

95R

elia

bilit

y =

Hig

h co

rrel

atio

n be

twee

n sw

ing

wei

ghts

,0.

64–0

.69,

but

rank

ing

and

rank

ord

er c

entr

oid

A v

alue

hie

rarc

hy (

sim

ilar

to

smal

l num

bers

the

budg

et p

ie w

here

100

poi

nts

wer

e al

loca

ted

betw

een

attr

ibut

es)

and

rank

ing

syst

ems

277

Cla

rk,1

974

Rep

orts

unp

ublis

hed

wor

k by

Ost

rom

,whi

chst

ates

mid

dle

and

uppe

r st

atus

res

pond

ents

Met

hodo

logi

cal p

aper

on

the

have

litt

le d

iffic

ulty

usi

ng t

his

appr

oach

.Aft

erbu

dget

pie

trai

ned

inte

rvie

wer

s ex

plai

ned,

resp

onse

rat

es

of 8

0–90

% h

ave

been

ach

ieve

d w

ith lo

wer

st

atus

res

pond

ents

282

Stra

uss

& H

ughe

s,19

76R

espo

nse

rate

to

a po

stal

que

stio

nnai

re w

asSm

all c

ost

28.5

% (

1001

/351

7) a

fter

1 r

emin

der

and

ain

volv

edTa

x ex

pend

iture

5-

wee

k cu

t-of

f.M

etho

d de

scri

bed

as(£

6000

) re

com

men

datio

ns b

y re

side

nts

“sim

ple

and

inex

pens

ive”

for

self-

of N

orth

Car

olin

a.Bu

dget

pie

ad

min

iste

red

met

hod

with

allo

catio

n of

qu

estio

nnai

reco

upon

s an

d bu

dget

con

stra

int

281

Rat

cliff

e,19

99R

espo

nse

rate

of 3

8% t

o a

conv

enie

nce

sam

ple

Inte

rnal

val

idity

con

firm

ed

Allo

catio

n of

don

or li

ver

graf

ts14

% t

houg

ht d

iffic

ult,

27%

mod

erat

ely

diffi

cult

Of 3

03 r

espo

nden

ts,2

exh

ibite

d do

min

ant

and

22%

slig

htly

diff

icul

tpr

efer

ence

s by

con

sist

ently

allo

catin

g al

l 10

0 liv

ers

to t

he g

roup

of i

ndiv

idua

ls w

ith

the

high

est

expe

cted

leng

th o

f sur

viva

l


143Allo

cati

on

of

poin

ts c

ontd

Prop

erti

es: b

ecau

se t

he

“bud

get”

of

poin

ts (

toke

ns,

mon

ey, l

ives

) to

allo

cate

is f

ixed

, res

pon

den

ts a

re s

ubje

cted

to

a co

nst

rain

ed c

hoi

ce.27

6,27

7,56

1G

iven

th

ew

ordi

ng

of t

he

ques

tion

, it

has

bee

n a

rgue

d to

pos

sess

car

din

al p

rope

rtie

s.27

7

Sch

edul

e fo

r th

e ev

alua

tio

n o

f in

divi

dual

qua

lity

of

life

– di

rect

wei

ghti

ng (

SE

IQo

L–D

W)

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

274

Hic

key

et a

l.,19

96Q

uick

and

pra

ctic

al in

clin

ic

Empi

rica

l app

licat

ion

of

SEIQ

oL–D

W t

o H

IV/A

IDS

275

Brow

ne e

t al

.,19

97M

ore

relia

ble

than

SEI

QoL

.DW

too

k r

= 0

.73

(SEI

QoL

)W

eigh

ts d

eriv

ed

Wei

ghts

pro

duce

d by

SEI

QoL

and

5

min

utes

to

unde

rsta

nd a

nd c

ompl

ete

by S

EIQ

oL–D

W

SEIQ

oL–D

W d

iffer

ed:a

t tim

e 1

the

mea

n D

evel

opm

ent

of t

he

com

pare

d w

ith 1

0–30

min

utes

for

SEIQ

oL

chan

ged

on a

vera

ge b

ydi

ffere

nce

was

7.8

poi

nts

and

at t

ime

SEIQ

oL–D

W p

roce

dure

(an

d(a

nd s

light

ly le

ss t

ime

7–10

day

s la

ter)

4.5

poin

ts c

ompa

red

2 7.

2 po

ints

a co

mpa

riso

n w

ith S

EIQ

oL)

to 8

.4 w

ith S

EIQ

oL

(7–1

0 da

ys la

ter)

R2

= 0

.78

(SEI

QoL

)

κ=

0.5

1 (S

EIQ

oL–D

W)

κ=

0.3

1 (S

EIQ

oL)

14–2

0 da

ys la

ter

55%

of

res

pond

ents

cou

ld

iden

tify

the

wei

ghts

th

ey p

rovi

ded

in

SEIQ

oL–D

W

565

Brow

ne e

t al

.,19

97SE

IQoL

–DW

was

ada

pted

for

use

(from

SEIQ

oL)

whe

n ra

pid

answ

ers

requ

ired

Con

cept

ual o

verv

iew

of q

ualit

y of

life

ass

essm

ent

incl

udin

g in

ter

alia

SEIQ

oL

Appendix 2

144 Allo

cati

on

of

poin

ts c

ontd

Prop

erti

es: b

ecau

se t

he

“bud

get”

of

poin

ts (

toke

ns,

mon

ey, l

ives

) to

allo

cate

is f

ixed

, res

pon

den

ts a

re s

ubje

cted

to

a co

nst

rain

ed c

hoi

ce.27

6,27

7,56

1G

iven

th

ew

ordi

ng

of t

he

ques

tion

, it

has

bee

n a

rgue

d to

pos

sess

car

din

al p

rope

rtie

s.27

7

Pat

ient

-gen

erat

ed in

dex

(PG

I)

Ref

.E

mpi

rica

l/met

hodo

logi

cal

Acc

epta

bilit

y to

res

pond

ents

Co

stIn

tern

al

Rep

rodu

cibi

lity

Val

idit

yst

udy

cons

iste

ncy

269

Rut

a et

al.,

1999

74%

res

pons

e ra

te (

571/

777)

.Of t

hese

,63%

Test

–ret

est

coef

ficie

ntH

igh

corr

elat

ion

with

a c

linic

al (

low

bac

k co

mpl

etio

n (3

59).

Pilo

t:47

% fu

lly a

nd=

0.7

at

2 w

eeks

pain

) m

easu

re a

nd S

F-36

Post

al P

GI f

or m

easu

ring

qua

lity

corr

ectly

com

plet

e(m

ore

relia

ble

than

of li

fe.C

ombi

nes

patie

nt s

elf-

SF-3

6 fo

r gr

oup

Ref

erre

d pa

tient

s ha

d si

gnifi

cant

ly lo

wer

as

sess

men

t an

d w

eigh

ting

com

pari

sons

)(w

orse

) sc

ore

than

tho

se m

anag

ed in

ge

nera

l pra

ctic

e

270

Mac

duff

& R

usse

ll,19

9881

% r

espo

nse

rate

(13

1/16

1):6

2% u

sabl

e G

ood

test

–ret

est

but

Box

6 is

inte

nded

to

rela

te p

robl

ems

to r

est

rate

at

time

1,52

% a

t tim

e 2

low

sam

ple

elig

ible

for

of li

fe a

s ba

sis

for

wei

ghtin

g.R

esul

ts s

ugge

stU

se o

f PG

I and

SF-

36 t

o re

test

ing

beca

use

ofth

at it

is n

ot u

nder

stoo

d by

res

pond

ents

and

mea

sure

pat

ient

qua

lity

if lif

e in

accu

rate

com

plet

ion

ther

efor

e th

e PG

I is

not

asse

ssin

g ov

eral

lin

a d

isab

led

popu

latio

n qu

ality

of l

ifeov

er t

ime

271

Rut

a et

al.,

1999

75.4

% r

espo

nse

rate

(13

17/1

746)

;Te

st–r

etes

t co

effic

ient

PGI c

orre

late

d w

ith 4

con

ditio

n-sp

ecifi

c 51

% (

672)

of t

hese

cor

rect

ly c

ompl

eted

= 0

.65

at 2

wee

kssc

ores

Met

hodo

logi

cal p

aper

eva

luat

ing

the

relia

bilit

y,va

lidity

and

re

spon

sive

ness

of t

he P

GI

in 4

con

ditio

ns

272

Her

d et

al.,

1997

Cor

rela

ted

‘mod

erat

ely

wel

l’ w

ithD

erm

atol

ogy

Life

Qua

lity

Inde

x an

d PG

IM

easu

rem

ent

of q

ualit

y of

life

co

rrec

tly s

tres

sed

the

wor

st p

robl

ems.

PGI

in a

topi

c de

rmat

itis

patie

nts

elic

ited

prob

lem

s no

t id

entif

ied

by s

truc

ture

dus

ing

PGI a

nd D

erm

atol

ogy

Der

mat

olog

y Li

fe Q

ualit

y In

dex,

and

scor

edLi

fe Q

ualit

y In

dex

qual

ity o

f life

wor

se t

han

did

Der

mat

olog

y Li

fe Q

ualit

y In

dex

273

Jenk

inso

n et

al.,

1998

89/1

00 c

ompl

etio

n w

ith P

GI a

nd

Euro

QoL

not

cor

rela

ted

with

SF-

36 o

r PG

IEu

roQ

oL;8

6/10

0 w

ith S

F-36

Com

pari

son

of 3

tec

hniq

ues

PGI b

y fa

r th

e m

ost

sens

itive

to

chan

gefo

r m

easu

ring

pat

ient

(se

lf-ov

er t

ime

repo

rted

) he

alth

sta

tus:

PGI,

Euro

QoL

and

SF-

36


145

Appendix 3

Additional data for chapter 6

We have provided a summary of some of the studies identified in the review to

provide the reader with a summary of the main

issues raised for each of the qualitative techniques.Note: this table does not provide a listing of all thestudies identified in our review.

Appendix 3

146 Inte

rvie

ws

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

291

Cra

btre

e &

Mill

er,1

991

4 pa

tient

s w

ith r

ecen

t ex

peri

ence

of p

ain

and

A p

urpo

sive

sam

ple

was

sel

ecte

d to

ens

ure

that

res

pond

ents

and

inte

rvie

wer

s w

ere

4 ph

ysic

ians

wer

e in

terv

iew

ed a

bout

pai

nst

rang

ers

and

that

res

pond

ents

did

not

hav

e a

spec

ialis

ed k

now

ledg

e of

the

top

ic.T

he t

ime

A q

ualit

ativ

e ap

proa

ch t

o pe

rcep

tion.

Thi

s pa

per

deta

ils t

he r

esea

rch

proc

ess

scal

e w

as c

onsi

dere

d to

be

acce

ptab

le a

nd t

he c

ost

was

con

side

red

in r

elat

ion

to t

he t

ime

prim

ary

care

res

earc

h:th

e in

clud

ing

ques

tionn

aire

dev

elop

men

t,tr

ansc

ript

ion,

need

ed t

o co

mpl

ete

the

stud

y to

a h

igh

stan

dard

.Eac

h re

sear

cher

wro

te d

own

thei

rlo

ng in

terv

iew

iden

tifyi

ng u

tter

ance

s an

d id

entif

ying

em

ergi

ng t

hem

esex

peri

ence

s of

pai

n in

ord

er t

o un

ders

tand

bia

ses

that

may

be

intr

oduc

ed r

athe

r th

an

to e

limin

ate

bias

.Als

o,3

rese

arch

ers

wer

e in

volv

ed w

ith in

depe

nden

tly r

eadi

ng in

terv

iew

sc

ript

s.T

he s

tudy

is s

tate

d as

bei

ng g

ener

alis

able

and

tra

nsfe

rabl

e

287

Fost

er,1

994

A n

umbe

r of

tea

cher

s an

d le

ctur

ers

wer

e ap

proa

ched

Adv

anta

ges

incl

ude

savi

ng t

ime

and

mon

ey b

ecau

se o

f the

elim

inat

ion

of t

rans

crip

tion

ofvi

a em

ail t

o es

tabl

ish

how

the

y go

abo

ut p

lann

ing

and

inte

rvie

w t

apes

,and

no

need

for

trav

el t

o pa

rtic

ipan

ts’ h

omes

or

offic

es.D

isad

vant

ages

are

Fish

ing

with

the

net

for

desi

gnin

g th

eir

cour

ses

that

mis

unde

rsta

ndin

gs c

anno

t be

inst

antly

cla

rifie

d,an

d in

tere

stin

g em

ergi

ng is

sues

can

not

rese

arch

dat

abe

pro

bed.

Als

o,m

ail m

ay b

e vi

ewed

as

bein

g a

nuis

ance

to

busy

pro

fess

iona

ls

289

Dic

ker

& A

rmst

rong

,199

516

pat

ient

s fr

om o

ne g

ener

al p

ract

ice

wer

e in

terv

iew

edT

he s

ampl

e w

as c

hose

n to

pro

vide

a c

ross

-sec

tion

of t

he t

ypes

of p

eopl

e at

tend

ing

the

for

30–4

0 m

inut

es,e

xplo

ring

und

erly

ing

assu

mpt

ions

prac

tice,

rath

er t

han

seek

ing

to b

e re

pres

enta

tive.

One

per

son

iden

tifie

d th

emes

from

the

Patie

nts’

vie

ws

of p

rior

ity

of a

ttitu

des

tow

ards

rat

ioni

ng in

the

NH

Str

ansc

ript

s.Pa

tient

s fo

und

it un

acce

ptab

le t

o di

scus

s is

sues

in r

elat

ion

to t

hem

selv

es a

ndse

ttin

g in

hea

lth c

are:

an

foun

d it

easi

er t

o co

nsid

er o

ther

peo

ple’

s ne

eds,

they

als

o fe

lt ill

equ

ippe

d to

ans

wer

inte

rvie

w s

urve

y in

pr

iori

ty-s

ettin

g qu

estio

ns in

gen

eral

one

prac

tice

290

Will

iam

s et

al.,

1998

15 p

eopl

e re

ferr

ed t

o a

com

mun

ity m

enta

l hea

lth t

eam

A s

mal

l sam

ple

of p

eopl

e pa

rtic

ipat

ed (

no e

xpla

natio

n of

sam

ple

size

or

recr

uitm

ent

isw

ere

inte

rvie

wed

bot

h be

fore

and

aft

er t

heir

initi

alpr

ovid

ed).

Att

entio

n is

pai

d to

the

loca

tion

of t

he in

terv

iew

s:it

was

tho

ught

tha

t co

nduc

ting

The

mea

ning

of p

atie

nt

cons

ulta

tions

to

esta

blis

h th

eir

satis

fact

ion

with

the

them

in t

he h

omes

of t

he p

atie

nts

was

adv

anta

geou

s.Se

cond

inte

rvie

ws

wer

e un

dert

aken

satis

fact

ion:

an e

xpla

natio

n se

rvic

e th

ey r

ecei

ved

so a

s to

pro

vide

an

oppo

rtun

ity fo

r re

spon

dent

val

idat

ion

(no

men

tion

of w

ho o

r ho

wof

hig

h re

port

ed le

vels

man

y pa

rtic

ipat

ed in

the

ana

lysi

s st

age)

292

Wils

on e

t al

.,19

9820

inte

rvie

ws

wer

e un

dert

aken

as

the

pilo

t st

udy

toTe

leph

one

inte

rvie

ws

wer

e pe

rcei

ved

to b

e th

e m

ost

appr

opri

ate

met

hod

for

the

mai

nas

cert

ain

if te

leph

one

or fa

ce-t

o-fa

ce in

terv

iew

s w

ould

stud

y.A

dvan

tage

s w

ere

resp

onse

rat

e,co

st-e

ffect

iven

ess,

less

tim

e ne

eded

for

each

Tele

phon

e or

face

-to-

face

be

app

ropr

iate

for

exam

inin

g co

ntin

ence

car

e.D

e Va

us’s*

inte

rvie

w,q

ualit

y of

res

pons

es a

nd in

terv

iew

er s

afet

yin

terv

iew

s?:a

dec

isio

n m

ade

5 co

nsid

erat

ions

wer

e us

ed t

o ai

d in

the

dec

isio

n-on

the

bas

is o

f a p

ilot

stud

ym

akin

g pr

oces

s

288

Aya

nian

et

al.,

1999

1392

pat

ient

s re

ceiv

ing

care

for

end

stag

e re

nal d

isea

seSa

mpl

e m

ay n

ot b

e re

pres

enta

tive

wer

e in

terv

iew

ed a

bout

pre

fere

nces

con

cern

ing

The

effe

ct o

f pat

ient

s’

tran

spla

ntat

ion

pref

eren

ces

on r

acia

l diff

er-

ence

s in

acc

ess

to r

enal

tr

ansp

lant

atio

n

* D

e Va

us D

A.Su

rvey

s in

soc

ial r

esea

rch.

3rd

ed.L

ondo

n:U

nive

rsity

Col

lege

Lon

don

Pres

s;19

91 (

cited

in W

ilson

and

col

leag

ues29

2 )


147Del

phi t

echn

ique

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

302

Tho

mso

n &

Pon

der,

1979

Use

d a

3-st

age

Del

phi t

o de

velo

p a

surv

ey t

echn

ique

The

stu

dy w

as p

ilote

d to

min

imis

e am

bigu

ities

,but

sel

ectio

n of

par

ticip

ants

was

ope

n to

for

use

in id

entif

ying

pri

oriti

es fo

r th

e Te

xas

Soci

ety

bias

es.D

ata

to b

e el

imin

ated

was

rep

orte

d an

d re

ason

s fo

r th

is p

rovi

ded.

Stat

ed a

s be

ing

Use

of D

elph

i met

hodo

logy

of

Alli

ed H

ealth

Pro

fess

ions

(T

SAH

P).2

1 he

alth

time-

cons

umin

g,an

d th

eref

ore

cost

ly p

lus

may

be

unac

cept

able

to

rese

arch

ers

into

gen

erat

e a

surv

ey

prof

essi

onal

s w

ere

recr

uite

d an

d w

ere

mai

nly

heal

thso

me

situ

atio

nsin

stru

men

t to

iden

tify

educ

ator

s or

adm

inis

trat

ors

prio

ritie

s fo

r st

ate

allie

d he

alth

ass

ocia

tions

297

Cha

rlto

n et

al.,

1981

Del

phi t

echn

ique

use

d by

the

Ken

t Are

a H

ealth

Asc

erta

ins

that

diff

eren

t pr

ofes

sion

als

adhe

re t

o th

eir

initi

al r

espo

nses

reg

ardl

ess

of b

eing

Aut

hori

ty t

o el

icit

spen

ding

pri

oriti

es o

f all

thos

eaw

are

of t

he o

pini

ons

of o

ther

gro

ups,

i.e.a

nony

mity

incr

ease

s va

lidity

? C

ost

is s

tate

d as

Spen

ding

pri

oriti

es in

Ken

t:in

volv

ed in

hea

lth s

ervi

ce d

ecis

ion-

mak

ing

(pro

fess

iona

lbe

ing

com

para

tivel

y lo

wa

Del

phi s

tudy

and

lay

polic

y-m

aker

s) in

Ken

t

300

Gab

bay

& F

ranc

is,1

988

2 pa

nels

(1

natio

nal c

onsi

stin

g of

9 p

anel

ists

,1 lo

cal

The

stu

dy is

eva

luat

ed in

the

dis

cuss

ion,

and

limita

tions

poi

nted

out

.It

high

light

s pr

oble

ms

cons

istin

g of

19

pane

lists

) w

ere

aske

d to

est

imat

e th

ew

ith g

ener

alis

abili

ty o

f the

nat

iona

l Del

phi s

tudy

.Sur

geon

s di

d no

t co

mm

ent

onH

ow m

uch

day

surg

ery?

pr

obab

le r

ates

of d

ay s

urge

ry fo

r 83

pro

cedu

res,

usin

gsu

bspe

cial

ist

oper

atio

ns,t

here

fore

the

res

ults

for

thes

e pr

oced

ures

may

be

less

rel

iabl

eD

elph

ic p

redi

ctio

nsD

elph

i.The

se r

esul

ts w

ere

then

com

pare

d w

ith e

ach

othe

r an

d w

ith a

Hos

pita

l Act

ivity

Ana

lysi

s

304

Burn

s et

al.,

1990

3-st

age

Del

phi a

sked

a v

arie

ty o

f hea

lth p

rofe

ssio

nals

Not

cle

ar h

ow m

any

disa

bilit

y or

gani

satio

n m

embe

rs w

ere

incl

uded

in t

he s

ampl

e.R

elia

bilit

yan

d m

embe

rs o

f dis

abili

ty o

rgan

isat

ions

to

esta

blis

hm

entio

ned

for

test

ing

corr

elat

ion

of r

esul

ts fo

r ro

unds

2 a

nd 3

,but

not

men

tione

d in

the

Prim

ary

heal

th c

are

need

s a

cons

ensu

s on

dis

abili

ty s

ervi

ce a

nd r

esea

rch

prio

ritie

sco

ntex

t of

pre

defin

ed c

rite

ria

of p

erso

ns w

ith p

hysi

cal

disa

bilit

ies:

wha

t ar

e th

e re

sear

ch a

nd s

ervi

ce

prio

ritie

s?

634

Silv

a &

Lew

is,1

991

A 3

-rou

nd D

elph

i tec

hniq

ue w

as u

sed

to a

sk 3

1 st

uden

t nu

rses

to

iden

tify

and

prio

ritis

e is

sues

of

Ethi

cs,p

olic

y,an

d al

loca

tion

ethi

cal c

once

rn t

hat

stem

from

the

allo

catio

n of

of s

carc

e re

sour

ces

in

scar

ce r

esou

rces

nurs

ing

serv

ice

adm

inis

-tr

atio

n:a

pilo

t st

udy

296

Har

ring

ton,

1993

Del

phi t

echn

ique

app

lied

to g

ain

opin

ions

of 5

3 se

nior

Sam

ple

limite

d by

tar

getin

g a

very

spe

cific

gro

up;i

mpr

ovem

ents

cou

ld b

e m

ade

by a

skin

gpr

actit

ione

rs c

once

rnin

g oc

cupa

tiona

l med

icin

em

ultid

isci

plin

ary

team

.Con

sens

us r

each

ed b

ut w

hen

chec

ked

by r

espo

nden

t va

lidat

ion

Res

earc

h pr

iori

ties

in

rese

arch

pri

oriti

esso

me

resu

lts w

ere

foun

d to

be

untr

ustw

orth

yoc

cupa

tiona

l med

icin

e:a

surv

ey o

f Uni

ted

Kin

gdom

m

edic

al o

pini

on b

y th

e D

elph

i tec

hniq

ue

cont

inue

d

Appendix 3

148 Del

phi t

echn

ique

con

td

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

305

Kas

tein

et

al.,

1993

The

pri

mar

y ai

m o

f thi

s st

udy

was

to

deve

lop

a se

t Ex

plor

es is

sues

of s

ituat

ion

and

pers

on-s

peci

fic b

iase

s,an

d m

akes

and

exp

lain

s m

etho

ds o

fof

cri

teri

a fo

r ev

alua

ting

the

perf

orm

ance

of f

amily

doin

g th

is fo

r th

eir

stud

y.D

oes

not

deal

with

all

type

s of

rel

iabi

lity

and

omits

to

exam

ine

Del

phi,

the

issu

e of

rel

iabi

lity:

phys

icia

ns c

onsu

lted

abou

t ab

dom

inal

pai

n.A

dditi

onal

ly,ot

her

crite

ria

such

as

valid

ity,a

ccep

tabi

lity

and

obje

ctiv

itya

qual

itativ

e D

elph

i stu

dy in

th

e se

cond

ary

obje

ctiv

e w

as t

o ex

plor

e is

sues

of

prim

ary

heal

th c

are

in

relia

bilit

y in

the

Del

phi t

echn

ique

the

Net

herl

ands

298

Rob

erts

et

al.,

1994

89 c

onsu

ltant

ger

iatr

icia

ns w

ere

aske

d to

sug

gest

No

inte

rnal

val

idat

ion.

Patie

nts

wer

e in

terv

iew

ed o

nly

once

;no

follo

w-u

p du

e to

mea

sure

s ap

prop

riat

e fo

r pe

rfor

man

ce in

dica

tors

in a

prac

tical

ities

of l

ocat

ing

the

patie

nts

Sett

ing

prio

ritie

s fo

r 2-

roun

d D

elph

i stu

dy a

nd 4

4 pa

tient

s w

ere

inte

rvie

wed

mea

sure

s of

per

form

ance

an

d as

ked

to r

ank

thes

e in

ord

er o

f im

port

ance

and

for

geri

atri

c m

edic

al s

ervi

ces

sugg

est

othe

r m

easu

res.

The

ran

k or

der

of t

he

2 gr

oups

is c

ompa

red

303

Gal

lagh

er e

t al

.,19

9628

exp

erts

in d

iabe

tes,

incl

udin

g pa

tient

s an

d pa

tient

Res

earc

her

bias

min

imis

ed e

nsur

ing

obje

ctiv

ity;i

nclu

sion

cri

teri

a fo

r pa

rtic

ipan

ts fo

rmul

ated

;re

pres

enta

tives

wer

e as

ked

to s

et p

rior

ities

for

atte

mpt

s w

ere

mad

e to

incl

ude

non-

resp

onde

rs.G

ener

al c

onsi

dera

tions

mad

e ab

out

Polic

y pr

iori

ties

in d

iabe

tes

impr

ovin

g ca

re fo

r di

abet

ic p

atie

nts,

usin

g a

2-ro

und

tech

niqu

e su

ch a

s co

nsen

sus

met

hods

ten

d to

igno

re d

isse

ntin

g co

ntri

butio

ns a

nd t

hat

poor

care

:a D

elph

i stu

dyD

elph

i sur

vey

sele

ctio

n of

par

ticip

ants

can

res

ult

in p

oor

relia

bilit

y

299

Had

orn

& H

olm

es,1

997

2-st

age

Del

phi p

roce

ss a

skin

g di

ffere

nt t

ypes

of

Incl

usio

n cr

iteri

a an

d pa

rtic

ipat

ing

num

bers

are

not

mad

e cl

ear;

this

is t

rue

of p

atie

nts

and

heal

th p

rofe

ssio

nals

to

set

crite

ria

for

elec

tive

mem

bers

of t

he p

ublic

as

wel

l as

for

prof

essi

onal

gro

ups.

It is

the

refo

re d

iffic

ult

to e

stab

lish

The

New

Zea

land

pri

ority

su

rgic

al p

roce

dure

sth

e va

lidity

,rel

iabi

lity

and

gene

ralis

abili

ty o

f thi

s ap

plic

atio

n of

the

tec

hniq

ue.I

t sh

ould

be

crite

ria

proj

ect.

Part

1:

stre

ssed

,how

ever

,tha

t th

is is

an

over

view

Ove

rvie

w

301

Wils

on &

Ker

r,19

984

Del

phi-s

tyle

sur

veys

wer

e co

nduc

ted

to g

ain

opin

ions

The

sam

ple

obta

ined

is s

tate

d as

not

bei

ng r

epre

sent

ativ

e:it

cons

iste

d of

par

ticip

ants

who

from

209

mem

bers

of a

bio

ethi

cs s

ocie

ty a

ndw

ere

wel

l edu

cate

d an

d af

fluen

t.T

his

limits

the

stu

dy in

ter

ms

of g

ener

alis

abili

ty.A

lso,

not

all

An

expl

orat

ion

of C

anad

ian

144

pers

ons

belie

ved

to b

e kn

owle

dgea

ble

abou

t th

ere

spon

dent

s re

plie

d to

all

4 st

ages

,alth

ough

num

bers

are

not

mad

e cl

ear

soci

al v

alue

s re

leva

nt t

o C

anad

ian

heal

thca

re s

yste

m a

bout

soc

ial v

alue

s in

heal

th c

are

rela

tion

to h

ealth

care

635

Cam

pbel

l et

al.,

1999

A 2

-rou

nd p

osta

l Del

phi s

urve

y in

volv

ing

both

Add

ress

es s

imila

r is

sues

as

abov

e.C

oncl

udes

tha

t pa

nel c

ompo

sitio

n ca

n in

fluen

ce t

he t

ypes

heal

thca

re m

anag

ers

and

fam

ily p

ract

ition

ers

on

of d

ecis

ion

mad

e,qu

estio

ning

the

rel

iabi

lity

of t

he m

etho

dT

he e

ffect

of p

anel

is

sues

of q

ualit

y of

pri

mar

y he

alth

care

mem

bers

hip

and

feed

back

on

rat

ings

in a

2-r

ound

D

elph

i sur

vey:

resu

lts o

f a

rand

omiz

ed c

ontr

olle

d tr

ial

295

Enda

cott

et

al.,

1999

The

Del

phi t

echn

ique

is u

sed

to g

ain

expe

rt o

pini

onR

epor

ts t

he m

etho

d as

hav

ing

stro

ng v

alid

ity a

nd n

o ev

iden

ce o

f rel

iabi

lity.

In t

his

(pae

diat

ric

inte

nsiv

e ca

re s

iste

rs)

of t

he n

eeds

of

appl

icat

ion

both

face

and

con

tent

val

idity

are

est

ablis

hed.

Dem

ande

d a

high

wor

kloa

d,C

an t

he n

eeds

of t

he c

ritic

ally

ch

ildre

n ad

mitt

ed t

o th

e in

tens

ive

care

uni

t.Sc

enar

ios

and

ther

efor

e is

not

acc

epta

ble

to a

ll th

ose

appr

oach

ed,w

hich

low

ered

the

res

pons

e ra

teill

chi

ld b

e id

entif

ied

usin

g ar

e pr

esen

ted

to t

he p

anel

and

are

ask

ed t

o in

dica

tesc

enar

ios?

Exp

erie

nces

of

the

impo

rtan

ce o

f eac

h ne

ed.T

he t

echn

ique

itse

lfa

mod

ified

Del

phi s

tudy

is e

valu

ated


149Focu

s gr

oup

s

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

320

Kel

ler

et a

l.,19

8722

indi

vidu

als

aged

65

and

over

,and

16

peop

le a

ged

A ‘n

on-p

roba

bilit

y’ s

ampl

e w

as u

sed.

Tele

phon

e fo

llow

-ups

wer

e ca

rrie

d ou

t to

giv

e th

ebe

twee

n 22

and

40

wer

e as

ked

to e

xpre

ss t

heir

vie

ws

oppo

rtun

ity fo

r pa

rtic

ipan

t fe

edba

ck t

o va

lidat

e w

hat

they

had

sai

d du

ring

the

gro

ups.

It is

Ass

essi

ng b

elie

fs a

bout

and

on

the

hea

lthca

re n

eeds

of o

lder

peo

ple.

The

sec

ond

uncl

ear

who

und

erto

ok t

he a

naly

sis

stag

e an

d if

mor

e th

an o

ne r

esea

rche

r w

as in

volv

edne

eds

of s

enio

r ci

tizen

s us

ing

grou

p w

as s

elec

ted

beca

use

they

wer

e pe

rcei

ved

to b

eth

e fo

cus

grou

p in

terv

iew

:kn

owle

dgea

ble

abou

t th

e st

ruct

ure

and

dyna

mic

s of

a qu

alita

tive

appr

oach

the

com

mun

ities

con

cern

ed

314

Kitz

inge

r,19

9435

1 pe

ople

par

ticip

ated

in o

ne o

f 52

focu

s gr

oups

.G

roup

s w

ere

pre-

exis

ting

to e

stab

lish

how

peo

ple

talk

abo

ut A

IDS

with

pee

rs,i

.e.h

ow t

hey

Look

ed a

t di

ffere

nt g

roup

s’ p

ersp

ectiv

es o

n A

IDS

mig

ht n

atur

ally

dis

cuss

thi

s is

sue.

Ever

yone

was

enc

oura

ged

to t

alk.

Thi

s w

as a

chie

ved

byT

he m

etho

dolo

gy o

f foc

us

play

ing

a ga

me

at t

he b

egin

ning

of e

ach

sess

ion

grou

ps:t

he im

port

ance

of

inte

ract

ion

betw

een

rese

arch

par

ticip

ants

312

Kud

er &

Roe

der,

1995

46 in

divi

dual

s (1

9 m

ale

and

27 fe

mal

e) t

ook

part

inG

ende

r an

d ra

ce w

ere

not

cons

ider

ed w

hen

sele

ctin

g gr

oup

mem

bers

.Ana

lysi

s an

d co

ding

focu

s gr

oup

disc

ussi

ons.

The

y w

ere

grou

ped

by a

ge a

ndw

ere

carr

ied

out

by m

ore

than

one

mem

ber

of s

taff

Att

itude

s to

war

d ag

e-ba

sed

soci

o-ec

onom

ic s

tatu

s.T

he g

roup

s w

ere

pres

ente

dhe

alth

car

e ra

tioni

ng:a

w

ith a

ser

ies

of s

cena

rios

and

ask

ed h

ow t

hey

wou

ldqu

alita

tive

asse

ssm

ent

dist

ribu

te s

carc

e re

sour

ces

317

Pow

ell e

t al

.,19

96A

sks

both

use

rs a

nd p

rovi

ders

of m

enta

l hea

lth

The

use

r gr

oups

wer

e se

lf-se

lect

ing,

and

ther

efor

e im

port

ant

view

s of

less

con

fiden

t pe

ople

serv

ices

in 4

sep

arat

e fo

cus

grou

ps t

o ge

nera

te id

eas

may

hav

e be

en m

isse

d.A

lso,

they

wer

e no

t st

ratif

ied

for

vari

able

s su

ch a

s ag

e,se

x an

dFo

cus

grou

ps in

men

tal

for

wha

t sh

ould

be

incl

uded

in q

uest

ionn

aire

s an

d to

oc

cupa

tion.

The

pro

vide

r gr

oups

con

sist

ed o

f sta

ff of

equ

ival

ent

leve

ls s

o as

not

to

behe

alth

res

earc

h:en

hanc

ing

valid

ate

wha

t is

in e

xist

ing

ques

tionn

aire

sin

timid

atin

g,bu

t al

so m

ay h

ave

mis

sed

impo

rtan

t in

form

atio

n.2

rese

arch

ers

wer

e in

volv

ed

the

valid

ity o

f use

r an

d in

the

ana

lysi

s st

age.

Stat

es t

he s

tudy

has

lim

ited

gene

ralis

abili

ty.T

he a

utho

rs d

iscu

ss a

llpr

ovid

er q

uest

ionn

aire

slim

itatio

ns o

f the

stu

dy

319

Stev

ens,

1996

13 le

sbia

n w

omen

wer

e as

ked

abou

t th

eir

heal

thca

reFr

ee fl

owin

g di

scus

sion

occ

urre

d,en

ablin

g to

pics

to

be d

ealt

with

in d

epth

.Onl

y th

ose

who

expe

rien

ces

and

thei

r in

tera

ctio

ns w

ith h

ealth

wer

e ve

ry w

ell e

duca

ted

wer

e in

clud

ed a

nd t

here

fore

yie

ldin

g a

bias

sam

ple.

Ana

lysi

s ca

rrie

dFo

cus

grou

ps:c

olle

ctin

g pr

ofes

sion

als

out

by t

he fa

cilit

ator

of t

he g

roup

sag

greg

ate

leve

l dat

a to

un

ders

tand

com

mun

ity

heal

th p

heno

men

a

325

Wei

nber

ger

et a

l.,19

9810

1 he

art

dise

ase

patie

nts

and

29 h

eart

phy

sici

ans

wer

eIt

was

foun

d th

at c

onsi

sten

cy b

etw

een

the

rate

rs w

as d

iffic

ult

to a

chie

veas

ked

abou

t th

eir

expe

rien

ce o

f hea

lthca

re a

nd fa

ctor

sC

an r

ater

s co

nsis

tent

ly

affe

ctin

g de

cisi

on-m

akin

g,re

spec

tivel

y.3

rate

rs w

ere

eval

uate

the

con

tent

of

aske

d to

cat

egor

ise

lists

form

ulat

ed fr

om t

he t

rans

crip

tsfo

cus

grou

ps?

so a

s to

est

ablis

h th

e le

vel o

f agr

eem

ent

betw

een

them

cont

inue

d

Appendix 3

150 Focu

s gr

oup

s co

ntd

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

322

Brad

ley

et a

l.,19

99C

ondu

cted

in 4

gen

eral

pra

ctic

es a

nd 1

sch

ool.

Ana

lysi

s st

age

carr

ied

out

by 2

inde

pend

ent

qual

itativ

e re

sear

cher

s.C

ost

stat

ed a

s be

ing

Incl

uded

173

peo

ple

in 2

4 fo

cus

grou

ps a

nd a

sked

‘inex

pens

ive’

at

£365

per

90-

min

ute

focu

s gr

oup.

Part

icip

ants

wer

e ab

le t

o gr

asp

them

esT

he h

ealth

of t

heir

nat

ion:

abou

t ci

tizen

s’ a

ttitu

des

tow

ards

Eng

land

’s he

alth

and

foun

d it

acce

ptab

le t

o di

scus

s to

pics

put

forw

ard

how

wou

ld c

itize

ns d

evel

op

stra

tegy

Engl

and’

s he

alth

str

ateg

y?

313

Coh

en &

Gar

rett

,199

9A

sked

gro

ups

of p

eopl

e w

ith m

enta

l illn

ess

abou

tA

gain

,use

d pr

e-ex

istin

g gr

oups

who

wer

e us

ed t

o di

scus

sing

the

ir s

ituat

ions

.The

faci

litat

orpa

tient

/wor

ker

rela

tions

hips

.Use

s ex

ampl

es o

f foc

usw

as a

ble

to e

stab

lish

a tr

ustin

g re

latio

nshi

p w

ith t

he p

artic

ipan

ts,w

hich

ena

bled

gro

upBr

eaki

ng t

he r

ules

:a g

roup

gr

oups

to

dem

onst

rate

tha

t so

met

imes

sen

sitiv

e is

sues

mem

bers

to

feel

com

fort

able

and

tal

k op

enly

wor

k pe

rspe

ctiv

e on

focu

s ca

n be

exp

lore

dgr

oup

rese

arch

323

Dol

an e

t al

.,19

9960

peo

ple

wer

e se

lect

ed u

sing

str

atifi

ed r

ando

mA

ccur

ate

and

cons

ider

ed o

pini

on o

btai

ned

due

to t

he p

erio

d of

del

iber

atio

n pr

ovid

ed.T

hesa

mpl

ing.

The

aim

was

to

esta

blis

h ho

w m

uch

attit

udes

sam

e 2

rese

arch

ers

cond

ucte

d al

l mee

tings

to

ensu

re c

onsi

sten

cy a

nd t

hat

all r

elev

ant

Effe

ct o

f dis

cuss

ion

and

and

opin

ions

cha

nged

aft

er a

per

iod

of d

iscu

ssio

n an

dto

pics

wer

e co

vere

d ad

equa

tely.

The

mee

tings

wer

e al

l tap

e-re

cord

ed a

nd t

rans

crib

ed,

delib

erat

ion

on t

he p

ublic

’s de

liber

atio

n w

as a

llow

edal

thou

gh it

was

unc

lear

who

car

ried

thi

s ou

tvi

ews

of p

rior

ity s

ettin

g in

he

alth

car

e:fo

cus

grou

p st

udy

Cit

izen

s’ ju

ries

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

334

Dun

kerl

ey &

Gla

sner

,199

8D

escr

ibes

a c

itize

ns’ j

ury

that

too

k pl

ace

in W

ales

,Bo

th o

rgan

iser

s an

d ju

rors

com

men

ted

that

an

unre

pres

enta

tive

sam

ple

of p

eopl

e ha

d w

hich

look

ed a

t ne

w t

echn

olog

ies

avai

labl

e fo

rbe

en s

elec

ted.

All

but

one

had

left

sch

ool a

t th

e m

inim

um le

avin

g ag

e,fe

w w

ere

in fu

ll-tim

eEm

pow

erin

g th

e pu

blic

? ge

netic

tes

ting

for

com

mon

dis

orde

rsem

ploy

men

t,th

ere

was

no

repr

esen

tativ

e fr

om e

thni

c m

inor

ity g

roup

s;th

ere

was

als

o no

Citi

zens

’ jur

ies

and

the

new

spre

ad o

f age

.The

obj

ectiv

ity o

f the

mod

erat

or c

anno

t be

est

ablis

hed

but

is s

tate

d as

hav

ing

gene

tic t

echn

olog

ies

expe

rien

ce in

faci

litat

ing

smal

l gro

up d

iscu

ssio

n.It

was

felt

that

som

e of

the

juro

rs d

id n

ot

fully

und

erst

and

all o

f the

mat

eria

l tha

t w

as b

eing

dis

cuss

ed


151Co

nsen

sus

pane

ls

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

336

Stro

nks

et a

l.,19

975

pane

ls w

ere

chos

en:e

ach

one

sele

cted

to

repr

esen

tT

he p

anel

rep

rese

ntin

g th

e pu

blic

was

sel

ecte

d fr

om u

nive

rsity

stu

dent

s an

d ci

vil s

erva

nts

a di

stin

ct g

roup

.The

se g

roup

s w

ere

patie

nts,

the

publ

ic,

not

wor

king

in h

ealth

.Thi

s ca

nnot

be

said

to

be r

epre

sent

ativ

e of

the

pub

lic.T

he a

utho

rsW

ho s

houl

d de

cide

? G

Ps,s

peci

alis

ts a

nd h

ealth

insu

rers

.Pan

el m

embe

rsal

so s

tate

tha

t no

ne o

f the

se h

ad a

chr

onic

illn

ess

or h

andi

cap.

Aut

hors

als

o st

ate

that

tho

seQ

ualit

ativ

e an

alys

is o

f pan

el

wer

e as

ked

to p

rior

itise

10

diffe

rent

hea

lth s

ervi

ces

peop

le t

akin

g pa

rt h

ad n

eith

er a

dequ

ate

know

ledg

e of

the

sub

ject

mat

ter

nor

time

toda

ta fr

om p

ublic

,pat

ient

s,de

liber

ate

thes

e m

atte

rshe

alth

care

pro

fess

iona

ls,

and

insu

rers

on

prio

ritie

s in

hea

lth c

are

Pub

lic m

eeti

ngs

Ref

eren

ceS

tudy

Des

crip

tio

nM

etho

dolo

gica

l co

mm

ents

341

Gun

dry

& H

eber

lein

,198

4U

ses

exam

ples

of 3

pub

lic m

eetin

gs a

nd s

imul

tane

ous

Alth

ough

rep

rese

ntat

iven

ess

of p

ublic

mee

tings

can

not

be e

stab

lishe

d,op

inio

n ga

ined

at

the

surv

eys

to t

est

3 di

ffere

nt h

ypot

hese

s re

latin

g to

the

m

eetin

gs g

ener

ally

mat

ched

opi

nion

gai

ned

thro

ugh

cond

uctin

g w

ider

sur

veys

Do

publ

ic m

eetin

gs

likel

y re

pres

enta

tiven

ess

of t

hose

att

endi

ng t

he

repr

esen

t th

e pu

blic

?m

eetin

gs

343

Got

t &

War

ren,

1991

Seve

ral p

ublic

mee

tings

wer

e se

t up

and

rev

iew

edPa

rtic

ipat

ion

enco

urag

ed b

y pe

rson

ally

invi

ting

mem

bers

of t

he c

omm

unity

,hav

ing

a di

vers

era

nge

of p

rofe

ssio

nals

att

endi

ng a

nd b

y ho

ldin

g th

e m

eetin

g in

a w

ell-k

now

n bu

ildin

gN

eigh

bour

hood

hea

lth

foru

ms:

loca

l dem

ocra

cy

at w

ork


153

BackgroundCurrent systematic reviews of methodologies havetaken a qualitative approach, highlighting howinstruments perform against predefined criteria.This study represents an attempt to estimate theweights of such predefined criteria.

Methods

Following the identification of criteria for assessingthe methodological status of techniques, as definedin chapter 4, a choice-based CA exercise was con-ducted to establish both the weights of thesecriteria and also whether differences existed acrossdisciplines. The criteria included in the study, andtheir corresponding levels, are shown in Table 8.

The combinations of criteria and levels resulted in 2916 (22 × 36) possible definitions of instruments.The computer software package SPEED636 was usedto reduce this to 28. These 28 scenarios wererandomly paired into 14 choices. For each choicethe respondent had to choose between instrumentA and B (Figure 1). Two scenarios were presented:one dealing with an allocative efficiency question(eliciting public view concerning which one of threedifferent treatments should be allocated scarcefunds), the other with a technical efficiency ques-tion (eliciting the public’s view of alternative ways ofproviding an asthma clinic). Two of the 14 choices(one in each scenario) were selected to gauge theinternal consistency of the responses. In these twochoices, all the criteria levels for one instrumentwere ‘better’, meaning that the respondent shouldchoose that option. Respondents who failed bothtests of consistency were assumed to be answering‘irrationally’, or not taking the questionnaireseriously, and were dropped from the analysis.

The choice CA postal questionnaire (aftersuccessive piloting and modification) was sent to five distinct disciplines: HEs, health serviceresearchers (HSRs), health council members(HCMs), medical sociologists (MSs) and PHCs.

From the responses to the 14 choices for eachrespondent, the following equation was estimated:

∆V = α 1ACCEPT + α 2CHOICE + α 3COST + α 4INTERNAL + α 5REPROD + α 6STRENGTH + α 7THEORY + α 8VALIDITY + e

where ∆V is the change in utility (or benefit) of moving from instrument A to B, and theindependent variables are the differences in thelevels of the criteria of the two instruments, asdefined in Table 8. The α 1 to α 8 represent theweights of the criteria. These weights indicate

Appendix 4

Establishing the relative weights of methodological criteria when evaluating

research techniques

TABLE 8 Criteria and levels for the discrete choice CA study

Criteria Levels Coding

Acceptability to respondents Low 1– ACCEPT Medium 2

High 3

Whether a constrained choice No 0is offered – CHOICE Yes 1

Cost – COST Low 1Medium 2High 3

Internal consistency – Low 1INTERNAL Medium 2

High 3

Reproducibility – REPROD Low 1Medium 2High 3

Strength of preference – No 0STRENGTH Yes 1

Theoretical basis – THEORY No 0Yes – from 1other disciplineYes – from own 2discipline

Validity – VALIDITY Low 1Medium 2High 3

Appendix 4

154

the marginal change in overall utility, V, resultingfrom a marginal change in a given criterion. Thismarginal change is obviously dependent on theunit of measurement. So, for example, whilst thecoefficient on acceptability to respondents indicatesthe marginal change in benefit of moving from say‘medium’ to ‘high’, the coefficient on ‘constrainedchoice’ indicates the marginal change in benefit of moving from ‘no’ to ‘yes’. The e represents theunobservable error term in the model. Given thatindividuals provide multiple observations, a random effects probit model was used to analysethe responses. A general to specific approach wasadopted, with criteria excluded in a backwardstepwise fashion if they were not statistically signifi-cant at the 5% level. Analysis was carried out firston the full data set and then by discipline. For thesegmented analysis, the Chow-type Likelihood Ratiotest was used to investigate whether the weights for the different criteria differed according to thescenarios presented. If there was no difference thedata sets could be merged and jointly analysedaccording to discipline.

Results

From a total of 1227 questionnaires sent out, 690were returned in the required timescale (whichincluded one reminder). Of these, 144 werereturned uncompleted and seven gave internallyinconsistent responses. The remaining sampleframe consisted of 539 questionnaires, giving aresponse rate of 56%.Table 9 provides thebackground characteristics of respondents.

For the total sample all the criteria had theexpected sign and were significant at the 5% level (Table 10). This suggests that all the criteriaare important when choosing between techniques.The positive values of seven of the eight criteriaindicate that the higher these are in instrument B relative to A, the more likely the respondent is to choose instrument B. Similarly, the negativevalue of ‘cost’ indicates that the lower the cost, the more likely the individual is to choose theinstrument. These results are all what we wouldexpect, providing support for the internal validityof the discrete choice exercise. The weight of each criterion is indicated by the coefficient. InTable 10, the coefficients are reported in order of their relative importance. As mentioned above,it is important to consider the unit of measure-ment when interpreting weights. For the com-bined group a marginal change in validity is more important than a marginal change in anyother criteria. This is followed by acceptability to respondents and strength of preference. Theleast important criteria are internal consistencyand theoretical basis.

TABLE 9 Sample frame on which the analysis was based

Discipline Frequency Percentage (%)

Health economics 156 28.9

Health service research 61 11.3

Health council members 111 20.6

Medical sociology 99 18.4

Public health 112 20.8

Choice 1

Acceptability to respondents

Constrained choice

Cost

Internal consistency

Reproducibility

Strength of preference

Theoretical basis

Validity

Instrument A

High

Yes

Medium

High

High

No

Other discipline

Medium

Instrument B

Medium

Yes

Low

Low

Low

Yes

Own discipline

Low

FIGURE 1 Example of a choice set in the weighting exercise

Which option would you Prefer instrument A Prefer instrument Bprefer? (tick one box only)

The Chow-type likelihood ratio test did not rejectthe hypothesis of homogeneity (the weights are the same irrespective of the scenario) for all butthe sample of HEs. The data for the two scenarioscould therefore be merged for all groups exceptHEs. The specific results for the five disciplines are shown in Table 11, and Table 12 shows therankings derived from the weights. It is importantto remember the unit of measurement wheninterpreting these weights.

Discussion and conclusionThe results from this weighting exercise suggestthat the criteria identified for evaluating quanti-tative instruments (from the literature review inthe main body of the report) reflect the criteriathat methodologists see as important whenchoosing a technique. This finding supports the evaluation of the techniques identified (again, in the main body of the report) according to the predefined criteria.

The aggregated results show that all the criteria are important to methodologists when choosing atechnique. Only HEs made a distinction betweenthe scenarios. Given that HEs made up a largeproportion of the sample frame (Table 9), thisgroup may bias the overall results. Given this,analysis was carried out for the individual groups.When this was done, only one discipline (MSs)viewed all eight criteria as important. Acceptabilityto respondents and validity were constantly rankedhighly by all disciplines, except by PHCs for thelatter. Internal consistency and theoretical validity(which was rarely significant) were consistentlyranked the lowest and constrained choice and cost fluctuated amongst the groups.

It is acknowledged that the robustness of these results may be affected by the relatively


155

TABLE 10 Weights for the criteria

Criteria Coefficient p-value(weight)

Validity 0.4854 0.001Acceptability to respondents 0.4790 0.001Strength of preference 0.2348 0.001Reproducibility 0.2231 0.001Constrained choice 0.2149 0.001Cost –0.1769 0.001Internal consistency 0.1354 0.001Theoretical basis 0.0485 0.017

Number of observations 7346Log-likelihood function –3166.5537AICa 0.8643

a Refer to Table 3 footnote (page 60)

TABLE 11 Weights by discipline and scenario (for HEs only)

Criteria Coefficient (weight)

HE HSR HCM MS PHC

Scenario 1 Scenario 2

Acceptability to 0.4799** 0.5979** 0.3777** 0.4899** 0.4911** 0.6722**respondents

Constrained choice 0.6916** 0.0873* 0.1334* 0.1283** 0.5035** 0.7456**

Cost –0.2280** –0.2294** ns –0.2664** –0.1607* –0.5430**

Internal consistency 0.2364** 0.1942** ns 0.1240** 0.1840** 0.2416**

Reproducibility 0.3770** ns 0.3672** 0.2037** 0.2509** 0.3136**

Strength of preference 0.4624** 0.3734** 0.2993** 0.1307* 0.2076* ns

Theoretical basis ns 0.1170** ns ns 0.1232* 0.0897*

Validity 0.5017** 0.3954** 0.7959** 0.2528** 0.6412** 0.2317**

Number of 1074 1068 813 1524 1334 1519observations

Log-likelihood function –414.9798 –478.8963 –324.4939 –737.1041 –509.0744 –587.2561

AICa 0.7858 0.9099 0.8106 0.9765 0.7752 0.7824

a Refer to Table 3 footnote (page 60)

ns, not significant

** p < 0.01; * p < 0.05

Appendix 4

156

low response rate (56%), with possible biasesintroduced through non-response. There is also an issue about the complex task given theinclusion of eight attributes, the subjective inter-pretation of these ‘qualitatively’ defined attributesand the assumption of a linear additive model.Nevertheless, these findings are potentially usefulfor a number of reasons. First, they may help

explain why certain disciplines choose certainresearch techniques. Secondly, the estimatedweights could potentially be used to ‘score’instruments in terms of their methodologicalstatus. Therefore, future research could developthis approach to carrying out a methodologicalsystematic review, taking account of the abovelimitations of this study.

TABLE 12 Summary of the relative importance rankings obtained from the CA results

Criteria Aggregated Segmented by discipline

HE HSR HCM MS PHC

Scenario 1 Scenario 2

Acceptability to 2 3 1 2 1 3 2respondents

Constrained choice 5 1 7 5 6 2 1

Cost 6 7 4 ns 2 7 3

Internal consistency 7 6 5 ns 7 6 5

Reproducibility 4 5 ns 3 4 4 4

Strength of preference 3 4 3 4 5 5 ns

Theoretical basis 8 ns 6 ns ns 8 7

Validity 1 2 2 1 3 1 6


157

Appendix 5

Summary of criteria used in priority setting and priority-setting frameworks

Appendix 5

158 Are

aC

rite

ria

Pri

ori

ty-s

etti

ng f

ram

ewo

rkIn

volv

emen

t o

f pu

blic

vie

ws?

Engl

and:

City

and

•

Rob

ustn

ess

or t

he e

xten

t to

whi

ch t

he p

ropo

sal c

an b

e im

plem

ente

dA

2-s

tage

sco

ring

sys

tem

was

use

d w

here

by fi

rst

the

prop

osal

s w

ere

✔H

ackn

ey28

0,63

7(0

–3)

rank

ed a

ccor

ding

to

whi

ch r

espo

nded

gre

ates

t to

loca

l nee

ds•

Prom

otio

n of

equ

ity (

0–1)

•Ev

iden

ce o

f effe

ctiv

enes

s or

cos

t-ef

fect

iven

ess

(0–2

)T

hey

wer

e th

en s

core

d fu

rthe

r us

ing

the

crite

ria

(pos

sibl

e sc

ores

•C

olla

bora

tion

or in

tegr

atio

n w

ith p

rim

ary

care

(0–

3)sh

own

in b

rack

ets)

•Pr

iori

tised

by

the

CH

C (

0–1)

•Pr

iori

tised

by

loca

l GPs

(0–

1)W

eigh

ts w

ere

dete

rmin

ed b

y th

e di

rect

or o

f pub

lic h

ealth

in•

Oth

er p

ossi

ble

or m

ore

appr

opri

ate

sour

ces

of fu

ndin

g (0

–5)

cons

ulta

tion

with

pur

chas

ing

team

man

ager

s an

d m

embe

rs o

f the

he

alth

aut

hori

ty (

not

stat

ed h

ow w

eigh

ts w

ere

dete

rmin

ed)

Prop

osal

s ra

nked

by

mul

tiply

ing

the

scor

e by

the

wei

ghts

Engl

and:

•H

ealth

gai

nW

eigh

ts a

scer

tain

ed u

sing

an

allo

catio

n of

poi

nts

exer

cise

:?

Sout

ham

pton

637

•Eq

uity

100

poin

ts t

o al

loca

te a

cros

s 5

crite

ria

•Lo

cal a

cces

s•

Pers

onal

res

pons

ibili

tyPr

opos

als

then

sco

red

acco

rdin

g to

the

am

ount

of h

ealth

gai

n ex

pect

ed•

Cho

ice

(sco

re o

f 1–3

) an

d w

heth

er o

r no

t ot

her

crite

ria

wer

e m

et (

scor

e of

0 o

r 1)

Each

of t

he p

ropo

sals

was

the

n ra

nked

by

mul

tiply

ing

each

cri

teri

a’s w

eigh

t by

the

ass

ocia

ted

scor

e,an

d su

mm

ing

Scot

land

:Arg

yll

•Po

tent

ial h

ealth

gai

nSc

orin

g an

d w

eigh

ting

in a

sin

gle

stag

e.Ea

ch c

rite

rion

is a

lloca

ted

a ra

nge

✗an

d C

lyde

638

•Pr

even

tion

of il

l hea

lthof

pos

sibl

e sc

ores

.For

exa

mpl

e,cr

iteri

on A

may

be

allo

cate

d a

max

imum

•Q

ualit

y of

life

scor

e of

10

and

a m

inim

um s

core

of –

5,co

mpa

red

with

cri

teri

on B

whi

ch•

Equi

ty o

f acc

ess

may

hav

e a

rang

e fr

om 5

to

–5.B

y de

finiti

on,c

rite

rion

A is

mor

e im

port

ant

•A

ddre

ssin

g he

alth

sta

tus

ineq

ualit

ies

at p

opul

atio

n le

vel

than

B•

Expr

esse

d de

man

d,ap

prop

riat

enes

s•

Stre

ngth

of e

vide

nce

Hea

lth p

ropo

sals

are

the

n sc

ored

for

each

cri

teri

on a

ccor

ding

to

this

ran

ge,

•K

now

n pr

iori

ties

and

a to

tal s

core

is e

stim

ated

for

each

pro

posa

l•

Add

ition

al c

ost

per

pers

on r

ecei

ving

the

inte

rven

tion

Scot

land

:Ayr

shir

e •

Hea

lth g

ain

Each

of t

he c

rite

ria

is fi

rst

scor

ed o

ut o

f 10

and

then

mul

tiplie

d by

its

✔an

d A

rran

639

•Ef

fect

iven

ess

rela

tive

impo

rtan

ce (

wei

ghtin

g),w

hich

are

dec

ided

upo

n by

exe

cutiv

e an

d•

Equi

tyno

n-ex

ecut

ive

dire

ctor

s of

the

hea

lth b

oard

•Pu

blic

pre

fere

nce

•Fl

exib

ility

The

wei

ghts

are

det

erm

ined

by

an a

lloca

tion-

of-p

oint

s m

etho

d.•

Valu

e fo

r m

oney

Res

pond

ents

are

giv

en 6

0 po

ints

to

allo

cate

acr

oss

the

6 cr

iteri

a

Each

of t

he p

ropo

sals

are

the

n ra

nked

by

mul

tiply

ing

the

scor

e by

the

w

eigh

t an

d su

mm

ing

cont

inue

d

cont

inue

d


159Are

aC

rite

ria

Pri

ori

ty-s

etti

ng f

ram

ewo

rkIn

volv

emen

t o

f pu

blic

vie

ws?

Scot

land

:Gre

ater

•

Size

of h

ealth

gai

nA

s ab

ove,

each

of t

he c

rite

ria

are

scor

ed o

ut o

f 10

and

then

✗G

lasg

ow (

Wal

ker

•Q

ualit

y of

evi

denc

e to

sup

port

cha

nge

mul

tiplie

d by

the

ir w

eigh

ting,

whi

ch w

as a

gree

d up

on b

y a

A,p

erso

nal

•Fi

t w

ith lo

cal a

nd n

atio

nal p

rior

ities

colla

bora

tion

of h

ealth

boa

rd d

irec

tors

and

HEs

com

mun

icat

ion,

•En

hanc

e se

rvic

e qu

ality

(ot

her

than

hea

lth g

ain)

1999

)•

Faci

litat

e pr

ovis

ion

in p

rim

ary

care

The

wei

ghts

are

det

erm

ined

by

an a

lloca

tion-

of-p

oint

s m

etho

d.R

espo

nden

ts a

re g

iven

100

poi

nts

to a

lloca

te a

cros

s th

e 5

crite

ria

Rec

omm

ende

d sc

ores

ran

ging

from

0 t

o 10

wer

e as

sign

ed t

o po

ssib

le

leve

ls fo

r ea

ch o

f the

cri

teri

aEa

ch o

f the

pro

posa

ls a

re t

hen

rank

ed b

y m

ultip

lyin

g th

e sc

ore

by

the

wei

ght

and

sum

min

g

New

Es

sent

ial:

Usi

ng t

hese

cri

teri

a,m

embe

rs o

f the

Pub

lic H

ealth

Com

mis

sion

ass

esse

d✔

Zea

land

299,

640–

643

•th

e he

alth

issu

e ha

s a

sign

ifica

nt im

pact

on

the

curr

ent

and

futu

re

the

heal

th p

ropo

sals

,alth

ough

the

re is

no

men

tion

of q

uant

itativ

e va

lues

heal

th s

tatu

s of

the

pop

ulat

ion

assi

gned

to

each

of t

he c

rite

ria

•th

e he

alth

issu

e pr

omot

es p

opul

atio

n-ba

sed

met

hods

to

prot

ect/

prev

ent

heal

th

Hig

h w

eigh

ting:

•th

e he

alth

issu

e w

ill r

educ

e in

equa

litie

s in

hea

lth s

tatu

s•

the

heal

th is

sue

prom

otes

the

bes

t he

alth

gai

n fo

r th

e re

sour

ces

requ

ired

Med

ium

wei

ghtin

g:•

ther

e is

pub

lic s

uppo

rt fo

r th

e he

alth

issu

e

Scot

land

:•

Leve

l of e

vide

nce

of c

linic

al e

ffect

iven

ess

Con

sulta

nts

wor

king

with

in t

he h

ospi

tal t

rust

wer

e gi

ven

a ch

oice

-bas

ed✗

Abe

rdee

n R

oyal

•

Size

of h

ealth

gai

nqu

estio

nnai

re,a

nd t

he w

eigh

ts fo

r th

e cr

iteri

a w

ere

indi

rect

ly e

stim

ated

Hos

pita

ls T

rust

211

•C

ontr

ibut

ion

to p

rofe

ssio

nal d

evel

opm

ent

•C

ontr

ibut

ion

to e

duca

tion,

trai

ning

and

res

earc

hC

linic

al d

irec

tora

tes

wer

e th

en a

sked

to

scor

e th

eir

prop

osed

clin

ical

•St

rate

gy a

rea

serv

ice

deve

lopm

ents

acc

ordi

ng t

o ho

w w

ell t

hey

perf

orm

ed o

n ea

ch

of t

he c

rite

ria

The

se s

core

s w

ere

mul

tiplie

d by

the

wei

ghts

,and

a t

otal

sco

re e

stim

ated

fo

r ea

ch p

ropo

sal

Scot

land

:•

Evid

ence

of e

ffect

iven

ess

Prop

osal

s ar

e fir

st s

core

d ac

cord

ing

to e

ach

crite

rion

on

a 1–

5 sc

ale

✔D

umfr

ies

and

•Va

lue

for

mon

eyG

allo

way

644

•H

ealth

gai

n or

mai

nten

ance

Each

cri

teri

on is

giv

en a

wei

ght

betw

een

1 an

d 10

•Eq

uity

•M

atch

ing

a na

tiona

l pri

ority

or

boar

d pr

iori

tyEa

ch o

f the

pro

posa

ls a

re t

hen

rank

ed b

y m

ultip

lyin

g th

e sc

ore

by t

he

•Pu

blic

pre

fere

nces

wei

ght

and

sum

min

g

The

•

Is t

he c

are

nece

ssar

y fr

om t

he c

omm

unity

poi

nt o

f vie

w?

Prop

osal

s ha

d to

pas

s th

roug

h ea

ch o

f the

cri

teri

a be

fore

the

y co

uld

✔N

ethe

rlan

ds63

7•

If so

,has

it b

een

dem

onst

rate

d to

be

effe

ctiv

e?be

impl

emen

ted

•Is

it a

lso

effic

ient

,usi

ng s

uch

met

hods

as

QA

LYs?

•C

an it

stil

l be

left

to

indi

vidu

al r

espo

nsib

ility

?

cont

inue

d

Appendix 5

160

cont

inue

d

Are

aC

rite

ria

Pri

ori

ty-s

etti

ng f

ram

ewo

rkIn

volv

emen

t o

f pu

blic

vie

ws?

Ore

gon28

3–28

6,64

5–64

8Va

lue

to s

ocie

ty:

Com

mis

sion

ers

divi

ded

100

poin

ts b

etw

een

the

3 m

ain

crite

ria:

✔?

•pr

even

tion,

man

y be

nefit

s,im

pact

on

soci

ety,

qual

ity o

f life

,per

sona

l va

lue

to s

ocie

ty,v

alue

to

an in

divi

dual

nee

ding

the

ser

vice

and

resp

onsi

bilit

y,co

st-e

ffect

iven

ess,

com

mun

ity c

ompa

ssio

n,m

enta

l es

sent

ial t

o a

heal

thca

re p

acka

gehe

alth

and

che

mic

al d

epen

denc

yIn

all,

17 p

ropo

sals

wer

e th

en s

core

d ag

ains

t th

ese

3 cr

iteri

a on

aVa

lue

to a

n in

divi

dual

nee

ding

the

ser

vice

:sc

ale

of 1

–10

•pr

even

tion,

qual

ity o

f life

,abi

lity

to fu

nctio

n,le

ngth

of l

ife,m

enta

l he

alth

and

che

mic

al d

epen

denc

y,eq

uity

,effe

ctiv

enes

s of

tre

atm

ent,

The

wei

ghts

are

mul

tiplie

d by

the

rat

ing

scor

e to

der

ive

a to

tal

pers

onal

cho

ice

com

mun

ity c

ompa

ssio

nsc

ore

for

each

pro

posa

l

Esse

ntia

l to

basi

c he

alth

care

:•

prev

entio

n,m

any

bene

fits,

qual

ity o

f life

,cos

t-ef

fect

iven

ess,

impa

ct

on s

ocie

ty


161

Preferences for priority-setting criteria: developing a scoring andweighting system for health board/authorities

We are conducting a research project for the NHS R&D Health Technology Assessmentprogramme which is aimed at identifying the relative importance of criteria that arecommonly used to aid the priority-setting process. This study considers the potential for theapproach employed in the paper to be used to establish weights for the criterion such thata score can be developed which encompasses these.

This questionnaire is concerned with your personal/independent views and theimportance you place on the various criteria that are considered in this study. We wouldtherefore be grateful if you could take the time to fill in this questionnaire. Everyone’sresponses and opinions are important. Once it is completed it should be returned in theprepaid envelope (enclosed) by 24 August 1999. If you do not wish to respond pleasereturn it uncompleted.

All answers and data will remain confidential and will not be reported in ways that canidentify individual responses. If you have any queries about the study or the questionnaire,please contact Angela Bate at the Health Economics Research Unit, Department of PublicHealth, University of Aberdeen. Telephone 01224 663123, ext. 52783.

Thank you in advance for your assistance.

Angela Bate Mandy Ryan

Research Assistant MRC Senior Fellow

Appendix 6

Questionnaire

Appendix 6

162

This questionnaire contains questions relating to the importance of the criteria in the priority-setting process. Each one presents the criteria (defined below) that are commonly used toanalyse proposals and priorities. Care should therefore be taken to read the specific definitionand context within which these criteria and their corresponding levels are set. Whilst some ofthe criteria levels may be crude, they will provide useful information on the relative weights ofthe different criteria.

The questions are split into two sections under the headings scenario 1 and scenario 2. In eachcase you are first presented with questions which involve you having to choose between twotypes of proposal: A and B. Proposals A and B only differ with respect to the levels of the criteria(defined over the page), all other factors remain the same. For every question in these sectionswe would like you to choose between A and B. Following this, you are asked to evaluate theimportance of the criteria (as used in the priority-setting process) to you.

The criteria are set as below:

• potential health gain

• evidence of clinical effectiveness

• budgetary impact

• equity of access and health status inequalities

• quality of service

• community values and priorities.

☞(please turn over)


163

These can be further described in detail with their corresponding levels.

• Potential health gain

This takes into account the number of people that are affected, the effect of the proposal (i.e.whether it is potentially a life-saving intervention or potentially improves quality of life), andthe time span over which the health gain may occur. Note that health gain includes ill healthavoided (i.e. prevention):

life-saving now – life-saving intervention for the majority (~ 80%) of those affected now

sustained improvement now – significant sustained improvement in physical/mental health for majority now

sustained improvement later – significant sustained improvement in majority later

temporary improvement now – temporary improvement in majority now

temporary improvement later – temporary improvement in majority later.

• Evidence of clinical effectiveness

This is a measure of the quality of evidence used to support the clinical benefits/effectivenessand potential health gain for each of the proposals. In this exercise, it is assumed that thenature of the proposals mean that randomised controlled trials (RCTs) are possible. The levelshave been based on those put forward by SIGN (Scottish Inter-Collegiate Guidelines Network)and are widely recognised by consultants, GPs and managers in the Scottish health service:

MA – evidence obtained from meta-analysis (MA) or RCTs

RCT – evidence obtained from at least one RCT

descriptive – evidence obtained from at least one well-conducted clinical study but no RCT (e.g. controlled study without randomisation, quasi-experimental, descriptive)

expert opinion – evidence obtained from expert committee reports or opinions and/or clinical experience of respected authorities

none – no evidence available.

• Budgetary impact

Under this criterion, the additional cost that would be needed in order to implement the pro-posal is considered relative to the size of the current allocation to the specific disease/patientgroup in the examples (in this instance we are assuming a budget allocation of £1,000,000).Under this configuration, possible levels are:

big save – the proposal involves a cost saving (> £50,000)

small save – the proposal involves a cost saving (between £30,000 and £50,000)

none – no additional (or small additional) expense is required/saved, current resources are just reallocated within the disease area/patient group (between –£30,000 and +£30,000)

small expense – proposal involves small overall additional cost relative to what the disease area/patient group receives at present (between £30,000 and £50,000)

big expense – proposal involves large overall additional cost relative to what the disease area/patient group receives at present (> £50,000).

Appendix 6

164

• Equity of access and health status inequalities

Within these levels, the definition of ‘deprived’ is used to mean: ‘those who do not have thesame opportunity to use health services because of their ethnicity, social class, age etc.’, and‘remote’ refers to: ‘those who do not have the same opportunity to access health servicesbecause of where they live’. Possible levels therefore are:

big reduction in inequality – proposal targets remote and deprived areas exclusively

small reduction in inequality – proposal targets either remote or deprived areas exclusively

remains the same – no differentiation by deprivation or remoteness

small increase in inequality – proposal targets only non-deprived areas or those thatbenefit most from the service at present

big increase in inequality – proposal targets only non-deprived and non-remote areas.

• Quality of service

This refers to how the service is provided, which would be affected by the implementation ofthe proposal. In this case, the process quality is measured using the following criteria:

– meets waiting time targets

– conforms to local/national objectives (e.g. a move towards primary level care)

– incorporates continuity of care

– enables patients to be informed about their care.

The levels are:

two or more direct ‘hits’ – the proposal improves the acceptability of the way the service is delivered by fully addressing two or more of the aspects of quality as defined above

one direct ‘hit’ – the proposal improves the acceptability of the way the service is delivered by fully addressing one of the aspects of quality as defined above

two or more partial ‘hits’ – the proposal improves the acceptability of the way the service is delivered by partially addressing two or more of the aspects of quality as defined above

one partial ‘hit’ – the proposal improves the acceptability of the way the service is delivered by partially addressing one of the aspects of quality as defined above

no ‘hits’ – the nature of the proposal means that none of these aspects of quality is addressed.


165

• Community values and priorities

In this criterion, the opinions of the public, patients and carers, about the proposal, are takeninto consideration. The levels reflect the opinion (i.e. whether the proposal was supported orobjected to) and the quality of this information (i.e. the method used to collate theseopinions):

robust evidence ‘support’ – a collection of well-conducted studies or a research projectthat fully explores the opinions of community groups(public, patients, carers) using robust elicitation methods(e.g. jury mechanisms, several focus groups, large qualitysatisfaction surveys, or quantitative methods) which showsthat community preferences support the proposal

weak evidence ‘support’ – research using anecdotal evidence, small satisfaction surveys and small-scale quantitative surveys identify the preferences/opinions of some community members which indicates support for the proposal

robust evidence ‘indifferent’ – evidence collected using robust elicitation methods (as defined above) which shows that the community is divided over whether they support or object to the proposal

weak evidence ‘object’ – research using weaker evidence which indicates community objection to the proposal

robust evidence ‘object’ – research using robust evidence which shows that community preferences object to the proposal.

Appendix 6

166

Scenario 1

You have to imagine that you are comparing two proposals which have been put forward forimplementation. They both address NHSiS [NHS in Scotland] priorities in Scotland or theequivalent NHS Lead Priorities in England and Wales; in particular, they are concentrated withthe disease area of cancer (e.g. gynaecological cancer). The only way that the two proposalsdiffer from each other is through the levels that are associated with the criteria (as definedabove). You are now asked to make a choice between whether you prefer proposal A or B.Consider each choice separately and indicate your preference by ticking the appropriate box.Please tick one box for every choice: there are no right or wrong answers.

Section 1

Which proposal would you Prefer proposal A Prefer proposal Bprefer? (tick one box only)



Proposal A Proposal B

Potential health gain Temporary improvement Life-saving nowlater

Evidence of clinical effectiveness RCT MA

Budgetary impact Small save Small save

Equity of access and health status inequalities Small increase in Remains the sameinequality

Quality of service Two or more direct ‘hits’ One partial ‘hit’

Community values and priorities Robust evidence Robust evidence ‘object’‘indifferent’

• Choice 1


Potential health gain Temporaryimprovement Temporary improvementnow later

Evidence of clinical effectiveness None Descriptive

Budgetary impact Small expense None

Equity of access and health status inequalities Small increase in Big reduction in inequalityinequality

Quality of service One partial ‘hit’ One partial ‘hit’

Community values and priorities Weak evidence ‘object’ Weak evidence ‘support’

• Choice 2


Potential health gain Temporary improvement Life-saving nownow

Evidence of clinical effectiveness Descriptive Descriptive

Budgetary impact Small save Small expense

Equity of access and health status inequalities Big increase in inequality Small reduction in inequality

Quality of service Two or more partial ‘hits’ No ‘hits’

Community values and priorities Robust evidence in Robust evidencesupport ‘indifferent’

• Choice 3





167


Potential health gain Sustained improvement Life-saving nownow

Evidence of clinical effectiveness None None

Budgetary impact Small save Big save

Equity of access and health status inequalities Small reduction in Big reduction ininequality inequality

Quality of service One direct ‘hit’ Two or more direct ‘hits’

Community values and priorities Weak evidence Robust evidence‘support’ ‘support’

• Choice 4


Potential health gain Life-saving now Sustained improvement now

Evidence of clinical effectiveness Expert opinion RCT

Budgetary impact Big expense Small expense

Equity of access and health status inequalities Small increase in Big reduction ininequality inequality

Quality of service Two or more partial ‘hits’ Two or more partial ‘hits’

Community values and priorities Weak evidence Robust evidence‘support’ ‘object’

• Choice 5


Potential health gain Temporary improvement Sustained improvementnow later

Evidence of clinical effectiveness MA Expert opinion

Budgetary impact Big expense Small save

Equity of access and health status inequalities Big reduction in Big reduction ininequality inequality

Quality of service One direct ‘hit’ No ‘hits’

Community values and priorities Robust evidence Weak evidence‘indifferent’ ‘object’

• Choice 6

Appendix 6

168

Section 2

In this case, you are asked to imagine (for scenario 1) that you have been given a fixed budgetof 100 points, which you have to allocate to each of the criteria that have been describedthroughout this exercise:

importance

Potential health gain

Evidence of clinical effectiveness

Budgetary impact

Equity of access and health status inequalities

Quality of service

Community values and priorities

You should allocate points to the criteria from your given budget. All of the budget can beallocated to one criterion if you so wish and not all of the criteria have to receive points. Thenumber of points you give reflects the importance, in the priority-setting process, of thatcriterion to you. For example, if you allocate twice as many points to ‘potential health gain’compared to ‘quality of service’ this means that you think that the potential health gaincriterion is twice as important as the quality of service criterion when setting priorities.

Please ensure that before you leave this question you have allocated all 100 points and that the ‘importance’ column totals 100.

100


169

Scenario 2

You have to imagine that you are comparing two proposals which have been put forward forimplementation. Neither address NHSiS [NHS in Scotland] priorities in Scotland or theequivalent NHS Lead Priorities in England and Wales; in this case, they are concerned with apotential but not ‘big’ killer (e.g. asthma). The only way that the two proposals differ from eachother is through the levels that are associated with the criteria (as defined above). You are nowasked to make a choice between whether you prefer proposal A or B. Consider each choiceseparately and indicate your preference by ticking the appropriate box. Please tick one box forevery choice: there are no right or wrong answers.

Section 1





Potential health gain Sustained improvement Sustained improvementlater later

Evidence of clinical effectiveness Descriptive RCT

Budgetary impact Big save Big expense

Equity of access and health status inequalities Small increase in Small reduction ininequality inequality

Quality of service One direct ‘hit’ One partial ‘hit’

Community values and priorities Robust evidence ‘object’ Robust evidence ‘support’

• Choice 7


Potential health gain Sustained improvement Life-saving nowlater

Evidence of clinical effectiveness None RCT

Budgetary impact None None

Equity of access and health status inequalities Remains the same Big increase in inequality

Quality of service Two or more partial ‘hits’ One direct ‘hit’

Community values and priorities Robust evidence Weak evidence‘indifferent’ ‘object’

• Choice 8


Potential health gain Temporary improvement Temporary improvementnow later

Evidence of clinical effectiveness Expert opinion MA

Budgetary impact None Big save

Equity of access and health status inequalities Small reduction in Small reduction ininequality inequality

Quality of service Two or more direct ‘hits’ Two or more partial ‘hits’

Community values and priorities Robust evidence ‘object’ Weak evidence object

• Choice 9

Appendix 6

170






Potential health gain Sustained improvement Temporary improvementnow later

Evidence of clinical effectiveness MA Expert opinion

Budgetary impact None Small expense

Equity of access and health status inequalities Small increase in Remains the sameinequality

Quality of service Not relevant One direct ‘hit’

Community values and priorities Robust evidence ‘support’ Robust evidence ‘support’

• Choice 10


Potential health gain Temporary improvement Sustained improvementlater now

Evidence of clinical effectiveness None Expert opinion

Budgetary impact Big expense Big save

Equity of access and health status inequalities Big increase in inequality Big increase in inequality

Quality of service Not relevant One partial ‘hit’

Community values and priorities Robust evidence ‘object’ Robust evidence ‘object’

• Choice 11


Potential health gain Sustained improvement Sustained improvementnow later

Evidence of clinical effectiveness Descriptive MA

Budgetary impact Big expense Small expense

Equity of access and health status inequalities Remains the same Big increase in inequality

Quality of service Two or more direct ‘hits’ Two or more direct ‘hits’

Community values and priorities Weak evidence ‘object’ Weak evidence ‘support’

• Choice 12


Potential health gain Temporary improvement Life-saving nownow

Evidence of clinical effectiveness RCT Expert opinion

Budgetary impact Big save Big expense

Equity of access and health status inequalities Remains the same Small increase in inequality

Quality of service No ‘hits’ Two or more partial ‘hits’

Community values and priorities Weak evidence ‘support’ Weak evidence ‘support’

• Choice 13


171

Section 2

In this case, you are asked to imagine (for scenario 2) that you have been given a fixed budgetof 100 points, which you have to allocate to each of the criteria that have been describedthroughout this exercise:

importance

Potential health gain

Evidence of clinical effectiveness

Budgetary impact

Equity of access and health status inequalities

Quality of service

Community values and priorities

You should allocate points to the criteria from your given budget. All of the budget can beallocated to one criterion if you so wish and not all of the criteria have to receive points. Thenumber of points you give reflects the importance, in the priority-setting process, of thatcriterion to you. For example, if you allocate twice as many points to ‘potential health gain’compared to ‘quality of service’ this means that you think that the potential health gaincriterion is twice as important as the quality of service criterion when setting priorities.

Please ensure that before you leave this question you have allocated all 100 points and that the ‘importance’ column totals 100.

100

Appendix 6

172

Finally, could you please provide a few details about yourself? The information from the following questions will not be used to identify individuals but to assist the analysis. All answers will be treated as confidential.

1. What is your age?

2. What is your gender? Female Male

3. What is your job title? __________________________________________

4. a. Do you have ‘hands on experience’ of priority setting? Yes No

b. If ‘Yes’, at what level (trust, health board/authority)?

__________________________________________

c. And for how long have you being doing this type of work?

__________________________________________

5. Do you have experience of setting priorities using weighted criteria? Yes No

6. a. Do you think that weighted criteria are a usefulmeans by which to set priorities? Yes No

b. If ‘No’, give details:


173

7. Are there any other criteria that you would consider to be important for priority setting? (please list)

Thank you for taking the time to complete this questionnaire

If you would be willing to be contacted at a later date to discuss your views further pleasetick the relevant boxes:

I am willing to be contacted by telephone Yes No

Telephone number: ___________________________

The best time to contact me would be:

morning

afternoon

evening

anytime

Appendix 6

174

Are there any comments you would like to make regarding the questionnaire?

Please return your questionnaire in the envelope provided

☞


175

Again, thank you for completing the questionnaire. I would now like to do some follow-up from thequestionnaire and ask you some brief questions related to it.

The questionnaire in general1. Do you think that you were a relevant recipient for this questionnaire?

– Yes– No

The criteria2. Looking at each of the criteria in turn: did you understand all of the descriptions and terminology

(1 = didn’t understand; 2 = could only understand a little; 3 = understood most; 4 = understood all)?a. Potential health gainb. Evidence of clinical effectivenessc. Budgetary impactd. Equity of access and health status inequalitiese. Quality of servicef. Community values and priorities

3. Were any too brief, i.e. not explained enough?– Yes: which?– No

4. Were the descriptions accurate or precise enough (1 = none was very precise or accurate; 2 = somewere; 3 = most of them were; 4 = all very precise and accurate)?

5. Are there any additional criteria that you think were omitted from this section?

The scenariosThe scenarios were designed to incorporate other criteria that are of importance in decision-making, suchas whether the proposal fulfilled a national priority/strategy or was concerned with a big disease area. Do you think that this was achieved?– Yes– No

6. Did you understand all of the terms used in the scenarios (1 = didn’t understand any; 2 = could onlyunderstand some; 3 = understood most; 4 = understood all)?

7. Were they relevant?– Yes– No

8. Did you take them into account when answering the questions?– Yes– No

9. Were different criteria important under the two different scenarios?– Yes: which?– No

Appendix 7

Telephone interview schedule

Appendix 7

176

The paired choicesHow easy was it to decide between the choices (on a scale of 1–4: 1 = very difficult; 4 = very easy)?

10. Did you find the hypothetical proposals too hypothetical or were they realistic (on a scale of 1–4: 1 = very hypothetical/unrealistic; 4 = not at all hypothetical/very realistic)?

11. Did anything appear as, in reality, contradictory?– Yes: what?– No

12. How did you decide between the choices? Did you either:– consider all the criteria and ‘weigh-up’ the alternatives? Or– only concentrate on some criteria that you thought were important? What were these?

13. Was anything irrelevant or did some criteria become redundant when making your choices?– Yes: what?– No

14. Were there any choices that were particularly hard to make?– Yes: what?– No

The budget allocation questionHow easy was it to divide points between the various criteria (on a scale of 1–4: 1 = very difficult; 4 = very easy)?

15. How did you divide the points between the criteria (top, middle, bottom, all allocated the samepoints for example)?

16. Do you think that the number of points you gave something accurately reflected your strength ofpreference for that criteria, i.e. if you gave one 20 points and another 40 points, did this mean thelatter was twice as important to you?– Yes– No: why?

Comparing the two methods17. Was either method – paired choice or budget allocation:

– easier – which?– quicker – which?

The resultsThe results have shown that differences exist between the results collected using the two differentmethods. To test these methods could you:

18. Rank in order of importance in terms of priority decision-making (in your view) the criteriadescribed in the questionnaire?

19. Of the least important criteria, are any of them in fact irrelevant or not important at all – which?

20. Would you consider any of the criteria to be of equal importance (i.e. ranked the same) – which?

Priority setting in general21. Do you think that patients or the community have a role to play in priority setting?

– Yes: what?– No: why?


177

22. Do you think that the views of the public or patients are important when deciding whether or not toimplement a proposal (on a scale of 1–4: 1 = not at all important; 4 = very important)?

Future workFuture work is planned to investigate further the importance of these criteria in priority-setting decision-making contexts.

23. Do you have any comments for the future study?


185

Methodology Group

continued

Current and past membership details of all HTA ‘committees’ are available from the HTA website (see inside front cover for details)

Methodology ProgrammeDirectorProfessor Richard LilfordDirector of Research and DevelopmentNHS Executive – WestMidlands, Birmingham

ChairProfessor Martin BuxtonDirector, Health EconomicsResearch GroupBrunel University, Uxbridge

Professor Douglas AltmanProfessor of Statistics in MedicineUniversity of Oxford

Dr David ArmstrongReader in Sociology as Applied to MedicineKing’s College, London

Professor Nicholas BlackProfessor of Health Services ResearchLondon School of Hygiene & Tropical Medicine

Professor Ann BowlingProfessor of Health Services ResearchUniversity College LondonMedical School

Professor David ChadwickProfessor of NeurologyThe Walton Centre forNeurology & NeurosurgeryLiverpool

Dr Mike ClarkeAssociate Director (Research)UK Cochrane Centre, Oxford

Professor Paul DieppeDirector, MRC Health ServicesResearch CentreUniversity of Bristol

Professor Michael DrummondDirector, Centre for Health EconomicsUniversity of York

Dr Vikki EntwistleSenior Research Fellow, Health Services Research UnitUniversity of Aberdeen

Professor Ewan B FerlieProfessor of Public ServicesManagementImperial College, London

Professor Ray FitzpatrickProfessor of Public Health & Primary CareUniversity of Oxford

Dr Naomi FulopDeputy Director, Service Delivery & Organisation ProgrammeLondon School of Hygiene & Tropical Medicine

Mrs Jenny GriffinHead, Policy ResearchProgrammeDepartment of Health London

Professor Jeremy GrimshawProgramme DirectorHealth Services Research UnitUniversity of Aberdeen

Professor Stephen HarrisonProfessor of Social PolicyUniversity of Manchester

Mr John HendersonEconomic AdvisorDepartment of Health, London

Professor Theresa MarteauDirector, Psychology &Genetics Research GroupGuy’s, King’s & St Thomas’sSchool of Medicine, London

Dr Henry McQuayClinical Reader in Pain ReliefUniversity of Oxford

Dr Nick PayneConsultant Senior Lecturer in Public Health MedicineScHARRUniversity of Sheffield

Professor Joy TownsendDirector, Centre for Researchin Primary & Community CareUniversity of Hertfordshire

Professor Kent WoodsDirector, NHS HTAProgramme, & Professor of TherapeuticsUniversity of Leicester

Members


186

Programme DirectorProfessor Kent WoodsDirector, NHS HTAProgramme, &Professor of TherapeuticsUniversity of Leicester

ChairProfessor Shah EbrahimProfessor of Epidemiology of AgeingUniversity of Bristol

Deputy ChairProfessor Jon NichollDirector, Medical CareResearch UnitUniversity of Sheffield

Professor Douglas AltmanDirector, ICRF MedicalStatistics GroupUniversity of Oxford

Professor John BondDirector, Centre for Health Services ResearchUniversity of Newcastle-upon-Tyne

Ms Christine ClarkFreelance Medical WriterBury, Lancs

Professor Martin EcclesProfessor of Clinical EffectivenessUniversity of Newcastle-upon-Tyne

Dr Andrew FarmerGeneral Practitioner & NHS R&D Clinical ScientistInstitute of Health SciencesUniversity of Oxford

Professor Adrian GrantDirector, Health ServicesResearch UnitUniversity of Aberdeen

Dr Alastair GrayDirector, Health Economics Research CentreInstitute of Health SciencesUniversity of Oxford

Professor Mark HaggardDirector, MRC Institute of Hearing ResearchUniversity of Nottingham

Professor Jenny HewisonSenior LecturerSchool of PsychologyUniversity of Leeds

Professor Alison KitsonDirector, Royal College ofNursing Institute, London

Dr Donna LampingHead, Health ServicesResearch UnitLondon School of Hygiene & Tropical Medicine

Professor David NealProfessor of SurgeryUniversity of Newcastle-upon-Tyne

Professor Gillian ParkerNuffield Professor ofCommunity CareUniversity of Leicester

Dr Tim PetersReader in Medical StatisticsUniversity of Bristol

Professor Martin SeversProfessor in Elderly Health CareUniversity of Portsmouth

Dr Sarah Stewart-BrownDirector, Health ServicesResearch UnitUniversity of Oxford

Professor Ala SzczepuraDirector, Centre for HealthServices StudiesUniversity of Warwick

Dr Gillian VivianConsultant in NuclearMedicine & RadiologyRoyal Cornwall Hospitals TrustTruro

Professor Graham WattDepartment of General PracticeUniversity of Glasgow

Dr Jeremy WyattSenior FellowHealth KnowledgeManagement CentreUniversity College London

Members

HTA Commissioning Board

Current and past membership details of all HTA ‘committees’ are available from the HTA website (see inside front cover for details)

Copies of this report can be obtained from:

The National Coordinating Centre for Health Technology Assessment,Mailpoint 728, Boldrewood,University of Southampton,Southampton, SO16 7PX, UK.Fax: +44 (0) 23 8059 5639 Email: [email protected]://www.ncchta.org ISSN 1366-5278

Health Technology Assessm

ent 2001;Vol.5:No.5

Eliciting public preferences for healthcare

FeedbackThe HTA programme and the authors would like to know

your views about this report.

The Correspondence Page on the HTA website(http://www.ncchta.org) is a convenient way to publish

your comments. If you prefer, you can send your comments to the address below, telling us whether you would like

us to transfer them to the website.

We look forward to hearing from you.

Eliciting Public Preferences for Healthcare

Documents

Transcript of Eliciting Public Preferences for Healthcare