1

CADTH Value Methods Panel

Using Best Worst Scaling to elicit Values

Carlo Marra

Problem

2

?

Traditional DCEs• Discrete Choice Experiments increasingly used in health services research• Respondents choose a preferred specification of the good or service• Aim is to obtain quantitative estimates of utility (benefit) associated with different attribute levels describing the good or service

Example of a DCE

4

Best-Worst Scaling

• Devised by Finn & Louviere (JPPM 1992) – introduced to health care by McIntosh & Louviere

(HESG 2002) – statistical proof paper Marley & Louviere (J Math

Psych 2005) – ‘user guide’ by Flynn et al (JHE 2006)

• Differs from traditional DCEs in the nature of the choice task

• Individuals choose the best and the worst attribute based on the levels displayed in a given specification

Statistical issues• MNL is (usually) a first step

– Is there heterogeneity?– Likely covariates that characterise it?

• More complex methods?– Mixed logit– Latent class analysis

• Non/semi parametric

Problem

8

?

Analyzing results

• To get around the dreaded BLACK BOX• Best-minus-worst-scores• Easy to understand• Found to be linearly related to the ML estimates of

the conditional logit model in virtually every empirical study to date

• Scores can help guide analysis of choice data– eg. LCAs which may give spurious associations

9

Stratifying and Targeting Pediatric Medulloblastoma

10

Purpose

To determine preferences of the general population, parents and health professionals regarding trade-offs between treatment intensity and survival including test characteristics, functional outcomes, psychological outcomes and economic burden.

11

12

Parents:

• N=76 participants

Health professionals:

• N=193 participants

General population:

• N= 3006 participants

13

100% accuracy of test

95% accuracy of test

90% accuracy of test

85% accuracy of test

1. Accuracy of test:The possible levels of test accuracy in this survey are:

14

2. QoL/ Functional ability (Side effects of the radiotherapy): The possible health states in this survey are:

Child will have normal healthy life.

Child will experience mild disability.

Child will experience partial disability.

Child will experience severe disability.

3. Ten year survival rates:The possible levels of survival in this survey are:

15

Good prognosis

Intermediate prognosis

Poor prognosis

Baseline Survival Rate 90% 70% 40%

Levels 100% 85% 55%

95% 70% 40%

90% 55% 25%

80% 40% 10%

Best and Worst Survey Design Clinicians’ Survey

16

Best-Worst estimated parameters (paired model) for general public

 

good prognosis intermediate prognosis poor prognosis

 Attributes Estimate Prob Attributes Estimate Prob Attributes Estimate Prob

Accuracy of the

test

100% 2.15 <.0001 100% 3.81 <.0001 100% 4.32 <.0001

95% 1.08 <.0001 95% 3.12 <.0001 95% 3.57 <.0001

90% 0.52 <.0001 90% 2.45 <.0001 90% 2.89 <.0001

85% -0.26 <.0001 85% 1.89 <.0001 85% 2.37 <.0001

Quality of life

Normal life 2.97 <.0001 Normal life 4.35 <.0001 Normal life 4.92 <.0001

Mild disability -0.98 <.0001 Mild disability 1.07 <.0001 Mild disability 1.78 <.0001

Partial disability -1.58 <.0001 Partial disability 0.59 <.0001 Partial disability 1.29 <.0001

Severe disability -3.21 <.0001 Severe disability -1.28 <.0001 Severe disability -0.53 <.0001

Survival rate

100% 3.28 <.0001 85% 3.01 <.0001 55% 2.09 <.0001

95% 2.29 <.0001 70% 2.10 <.0001 40% 1.40 <.0001

90% 1.37 <.0001 55% 0.56 <.0001 25% 0.55 <.0001

80%0.00  

40%0.00  

10%0.00  

Good prognosisNumber of respondent 901

Attribute Times Shown

Times Selected

BestBest Count Proportion

Times Selected

Worst

Worst Count

Proportion

Best - Worst score

Accuracy of the

test

100% 3612 1307 36.2% 128 3.5% 117995% 3588 615 17.1% 214 6.0% 40190% 3597 283 7.9% 330 9.2% -4785% 3604 163 4.5% 702 19.5% -539

Quality of life

Normal life 3642 2094 57.5% 155 4.3% 1939Mild disability 3597 214 5.9% 1903 52.9% -1689

Partial disability 3605 141 3.9% 2822 78.3% -2681Severe disability 3604 88 2.4% 3162 87.7% -3074

Survival rate

100% 3596 2591 72.1% 117 3.3% 247495% 3596 1954 54.3% 145 4.0% 180990% 3595 913 25.4% 359 10.0% 55480% 3612 449 12.4% 775 21.5% -326

Baseline survival rate is 90%.

Best-Worst estimated parameters (paired model) for parents and clinicians – intermediate prognosis

  AttributesParents Clinicians

Estimate Prob Estimate Prob

Accuracy of the

test

100% 4.79 <.0001 3.49 <.0001

95% 4.25 <.0001 2.94 <.0001

90% 3.35 <.0001 2.14 <.0001

85% 3.11 <.0001 1.54 <.0001

Quality of life

Normal life 5.49 <.0001 3.67 <.0001

Mild disability 3.08 <.0001 1.38 <.0001

Partial disability 1.53 <.0001 0.90 <.0001

Severe disability -0.46 0.035 -0.68 <.0001

Survival rate

85% 4.44 <.0001 3.44 <.0001

70% 2.89 <.0001 2.19 <.0001

55% 0.76 0.003 0.61 <.0001

40%0.00   0.00  

• Normal life, 85% survival rate and 100% accuracy of the test are more favorable attributes for parents and clinicians.

• Severe disability is the only attribute that is less favorable than 40% survival rate.

• Comparing coefficients of mild disability for intermediate prognosis and good prognosis shows that parents and clinicians prefer mild disability over low of survival rate. For parents mild disability is more favorable than 70% survival rate.

Summary of results for clinicians in different prognosis

20

 

good prognosis intermediate prognosis poor prognosis

 

AttributesTimes shown

Times Selected

Best

Times Selected

WorstAttributes

Times shown

Times Selected

Best

Times Selected

WorstAttributes

Times shown

Times Selected

Best

Times Selected

Worst

Accuracy of

the test

100% 928 285 46 100% 913 475 41 100% 901 537 51

95% 930 145 60 95% 909 382 41 95% 901 466 53

90% 931 62 164 90% 908 157 105 90% 900 220 83

85% 926 39 276 85% 914 81 163 85% 898 125 127

Quality of life

Normal life 926 356 42 Normal life 914 508 42 Normal life 903 550 41

Mild disability 928 37 430 Mild disability 912 104 216 Mild disability 897 132 158

Partial disability 927 37 635 Partial disability 911 51 380 Partial disability 901 72 238

Severe disability 927 59 746 Severe disability 914 65 645 Severe disability 899 64 510

Survival rate

100% 929 736 28 85% 912 521 51 55% 901 275 130

95% 930 631 35 70% 910 277 140 40% 902 134 246

90% 928 288 93 55% 913 73 420 25% 899 53 508

80% 926 109 229 40% 914 42 492 10% 898 72 555

Best-Worst count score is equal to difference of times selected best and worst divided by times shown.

Best-Worst count score for clinicians’ preferences

• Quality of life has the most impact on clinicians’ decision making for good prognosis. Severe, partial and mild disability are least favorable attributes, respectively.

 

Good prognosisattributes

Best - Worst score

Intermediate

prognosisattributes

Best - Worst score

Poor prognosisattributes

Best - Worst score

Accuracy of the test

100% 25.8% 100% 47.5% 100% 53.9%

95% 9.1% 95% 37.5% 95% 45.8%

90% -11.0% 90% 5.7% 90% 15.2%

85% -25.6% 85% -9.0% 85% -0.2%

Quality of life

Normal life 33.9% Normal life 51.0% Normal life 56.4%

Mild disability -42.3% Mild disability -12.3% Mild

disability -2.9%

Partial disability -64.5% Partial

disability -36.1% Partial disability -18.4%

Severe disability -74.1% Severe

disability -63.5% Severe disability -49.6%

Survival rate

100% 76.2% 85% 51.5% 55% 16.1%

95% 64.1% 70% 15.1% 40% -12.4%

90% 21.0% 55% -38.0% 25% -50.6%

80%-13.0%

40%-49.2%

10%-53.8%

Summary

• Strengths– Easy– Simple to calculate - no

black box– Can be done online– Use of scores might give

average applied researcher more confidence in results

• Weaknesses– Does not meet

economists definition of a trade-off

– Cannot on its own produce QALYs