Ranking and Rating Data in Joint RP/SP Estimation

by

JD Hunt, University of CalgaryM Zhong, University of Calgary

PROCESSUS Second International Colloquium

Toronto ON, CanadaJune 2005

Overview• Introduction

• Context• Motivations

• Definitions• Revealed Preference Choice• Stated Preference Rankings• Revealed Preference Ratings• Stated Preference Ratings

• Estimation Testbed• Concept• Synthetic Data Generation• Results

• Conclusions

Overview• Introduction

• Context• Motivations

• Definitions• Revealed Preference Choice• Stated Preference Rankings• Revealed Preference Ratings• Stated Preference Ratings

• Estimation Testbed• Concept• Synthetic Data Generation• Results – so far

• Conclusions – so far

Introduction• Context

• Common task to estimate logit model utility function for non-existing mode alternatives

• Joint RP/SP estimation available• Good for sensitivity coefficients• Problems with alternative specific constants (ASC)

• Motivation• Improve situation regarding ASC• Seeking to expand on joint RP/SP estimation• Add rating information

• 0 to 10 scores• Direct utility

• Increase understanding of issues regarding ASC generally

Definitions• Revealed Preference Choice• Stated Preference Ranking• Revealed Preference Ratings• Stated Preference Ratings

• Linear-in-parameters logit utility function

Um = Σk αm,k xm,k + βm

Definitions• Revealed Preference Choice• Stated Preference Ranking• Revealed Preference Ratings• Stated Preference Ratings

• Linear-in-parameters logit utility function

Um = Σk αm,k xm,k + βmsensitivitycoefficient ASC

Revealed Preference Choice

• Actual behaviour• Best alternative choice from existing• Attribute values determined

separately• Indirect utility measure

– observe outcome

Umr = λr [ Σk αm,k xm,k + βm ] + βm

r

Revealed Preference Choice

• Disaggregate estimation provides

Umr = Σk α’m,k

r xm,k + β’mr

with

α’m,kr = λr αm,k

β’mr = λr βm + βm

r

Stated Preference Ranking

• Stated behaviour• Ranking alternatives from

presented set• Attribute values indicated• Indirect utility measure – observe

outcome

Ums = λs [ Σk αm,k xm,k + βm ] + βm

s

Stated Preference Ranking

• Disaggregate (exploded) estimation provides

Ums = Σk α’m,k

s xm,k + β’ms

with

α’m,ks = λs αm,k

β’ms = λs βm + βm

s

Revealed Preference Ratings

• Stated values for selected and perhaps also unselected alternatives

• Providing 0 to 10 score with associated descriptors

10 = excellent; 5 = reasonable; 0 = terrible

• Attribute values determined separately

• Direct utility measure (scaled?)

Rmg = θ

g [ Σk αm,k xm,k + βm ] + βmg

Revealed Preference Ratings

• Regression estimation provides

Rmg = Σk α’m,k

g xm,k + β’mg

with

α’m,kg = θ

g αm,k

β’mg = θ

g βm + βmg

Stated Preference Ratings• Stated values for each of set of

alternatives• Providing 0 to 10 score with

associated descriptors 10 = excellent; 5 = reasonable; 0 = terrible

• Attribute values indicated• Provides verification of rankings• Direct utility measure (scaled?)

Rmh = θ

h [ Σk αm,k xm,k + βm ] + βmh

Stated Preference Ratings

• Regression estimation provides

Rmh = Σk α’m,k

h xm,k + β’mh

with

α’m,kh = θ

h αm,k

β’mh = θh βm + βm

h

Estimation Testbed

• Specify true parameter values (αm,k and βm)• Generate synthetic observations

• Assume attribute values and error distributions• Sample to get specific error values• Calculate utility values using attribute values,

true parameter values and error values• Develop RP choice observations and SP ranking

observations using utility values• Develop RP ratings observations and SP ratings

observations by scaling utility values to fit within 0 to 10 range

• Test estimation techniques in terms of returning to true parameter values

True Utility Function

Um = Σk αm,k xm,k + βm + em

Alternative (m)

αm,k=1 αm,k=2 αm,k=3 αm,k=4 βm

1 -0.25 -0.50 -0.25 -0.50 0 2 -0.80 -0.50 -0.15 -0.40 -0.75 3 -0.65 -0.20 -0.55 -0.30 2.50 4 -0.50 -1.20 -0.70 -0.20 0 5 -0.40 -0.50 -0.15 -0.4 1.5 6 -0.05 -0.30 -0.50 -0.55 -0.80 7 -0.25 -0.50 -0.10 -0.80 1.50

True Parameter Values

Attribute Values

Alternative (m)

μm,k=1 σm,k=1 μm,k=2 σm,k=2 μm,k=3 σm,k=3 μm,k=4 σm,k=4

1 10.0 0.25 5.0 0.50 20.0 3.25 15.0 0.40

2 5.0 0.50 5.0 0.50 25.0 7.00 20.0 4.10

3 15.0 1.20 2.0 0.20 19.0 2.50 15.0 2.20

4 10.0 2.20 10.0 4.00 20.0 5.00 12.0 1.80

5 10.0 2.50 8.0 2.00 16.0 3.00 10.0 1.50

6 15.0 2.30 7.0 1.30 15.0 2.00 15.0 2.70

7 15.0 1.50 5.0 1.00 14.0 1.20 25.0 3.50

sampled from N(μm,k ,σm,k) with

Error ValuesSampled from N(μ= 0 , σ m )

σ m varies by observation type:

• RP Choice: σ m = σ rm = 2.4

• SP Rankings: σ m = σ sm =

1.5

• RP Ratings: σ m = σ gm =

2.1

• SP Ratings: σ m = σ hm = 1.8

Generated Synthetic Samples

• Each of 4 observation types• 7 alternatives for each observation

(m=7)• Set of 15,000 observations• Sometimes considered subsets of

alternatives with overall across observation types, as indicated below

Testbed Estimations

• RP Choice• SP Rankings• Joint RP/SP Data• Ratings• Combined RP/SP Data and Ratings

RP Choice

• Used ALOGIT software

• Set β’m=1r = 0 to avoid over-

specification• Provides:

• α’m,kr = λr αm,k

• β’mr = λr βm + βm

r

• Know that λr = π / ( √6 σ rm )

= 0.534

RP Choice Estimated vs True Valueswith 15,000 Observations

-18

-12

-6

0

6

12

18

-18 -12 -6 0 6 12 18

observed

estim

ated

RP Choice Estimates vs True Values

with 15,000 Observations

-2

-1.5

-1

-0.5

0

-2 -1.5 -1 -0.5 0

observed

estim

ated

RP Choice Estimated vs True Valueswith 15,000 Observations

-18

-12

-6

0

6

12

18

-18 -12 -6 0 6 12 18

observed

estim

ated

1

2

3

4 56

7

(2.3)(1.6)

(1.2)(0.6)

(0.8)

(1.1)

ρ20= 0.1834

ρ2c= 0.6982

RP ChoiceSelection frequencies and ASC estimates

Alternative (m)

Times Selected

Estimated ASC t-ratio

1 1,377 0 - 2 835 -3.63 1.13 53 13.94 2.34 25 -3.49 0.65 12,100 -3.79 1.26 594 -2.57 0.87 16 15.21 1.6

RP Choice 2Selection frequencies and ASC estimates

Alternative (m)

Times Selected

Estimated ASC t-ratio

1 1,051 0 - 2 894 -0.74 0.23 3,046 0.7 0.24 236 0.04 0.05 6,331 1.61 0.56 2,271 2.92 0.87 1,171 1.04 0.3

RP Choice 2 Estimated vs True Valueswith 15,000 Observations

-4

-3

-2

-1

0

1

2

3

4

-4 -3 -2 -1 0 1 2 3 4

observed

estim

ated

1

2

3

4

5

6

7

(0.2)(0.3)

(0.5)

(0.0)

(0.8)

(0.2)

ρ20= 0.3151

ρ2c= 0.1683

RP Choice 3Selection frequencies and ASC estimates

Alternative (m)

Times Selected

Estimated ASC t-ratio

1 1,682 0 - 2 1,783 -0.05 0.03 2,842 2.25 0.84 3,044 0.01 0.05 1,944 2.07 0.76 1,940 -0.12 0.07 1,765 2.6 0.9

RP Choice 3 Estimated vs True Valueswith 15,000 Observations

-4

-3

-2

-1

0

1

2

3

4

-4 -3 -2 -1 0 1 2 3 4

observed

estim

ated

1

2

3

4

5

6

7

(0.8)

(0.9

(0.7)

(0.0)(0.0)

(0.0)

ρ20= 0.1852

ρ2c= 0.1736

SP Rankings

• Used ALOGIT software

• Set β’m=1s = 0 to avoid over-

specification• Provides:

• α’m,ks = λs αm,k

• β’ms = λs βm + βm

s

• Know that λs = π / ( √6 σ sm )

= 0.855

SP Ranking Estimates vs True Valueswith 15,000 observations

-4

-3

-2

-1

0

1

2

3

4

-5 -4 -3 -2 -1 0 1 2 3 4 5

observed

estim

ated

observed

est

imate

d

86420-2-4-6-8

8

6

4

2

0

-2

-4

-6

-8

0

0

20 Runs for RP Choice Estimated vs True Values with 15,000 Observations

SP Rankings

• More information with full ranking• Also confirm against RP above

• ‘ranking version’ available• estimate using full ranking

RP Rankings Estimates vs True Valueswith 15,000 observations

-4

-3

-2

-1

0

1

2

3

4

-5 -4 -3 -2 -1 0 1 2 3 4 5

observed

estim

ate

d

SP vs RP Rankings

• ASC translated en bloc to some extent

SP Rankings: Role of σm,k

• Impact of changing σm,k used when synthesizing attribute values

• Sampling from N(μm,k ,σm,k)

• Different σm,k means different spreads on attribute values

• Impacts relative size of σ sm

• Implications for SP survey design

Attribute Values

Alternative (m)

μm,k=1 σm,k=1 μm,k=2 σm,k=2 μm,k=3 σm,k=3 μm,k=4 σm,k=4

1 10.0 0.25 5.0 0.50 20.0 3.25 15.0 0.40

2 5.0 0.50 5.0 0.50 25.0 7.00 20.0 4.10

3 15.0 1.20 2.0 0.20 19.0 2.50 15.0 2.20

4 10.0 2.20 10.0 4.00 20.0 5.00 12.0 1.80

5 10.0 2.50 8.0 2.00 16.0 3.00 10.0 1.50

6 15.0 2.30 7.0 1.30 15.0 2.00 15.0 2.70

7 15.0 1.50 5.0 1.00 14.0 1.20 25.0 3.50

sampled from N(μm,k ,σm,k) with

0.0001

0.001

0.01

0.1

1

10

0.1

0·σ

m,k

0.5

0·σ

m,k

1.0

0·σ

m,k

2.0

0·σ

m,k

4.0

0·σ

m,k

10

.00

·σm

,k

Factoring on σm,k

Av

era

ge

Ab

so

lute

Dif

fere

nc

e

Es

tim

ate

d a

nd

Tru

e V

alu

es α

β

SP Rankings: Role of σm,k

• Increasingσm,k improves estimators• Roughly proportional

• Ratio of βm toαm,k maintained

• Use 1.00 ·αm,k in remaining work here

• Implications for SP survey design• More variation in attribute values is

better

Joint RP/SP Data• Two basic approaches for αm,k

• Sequential (Hensher)

First estimate α’m,ks using SP observations;

Then estimate α’m,kr using RP observations,

also forcing ratios among α’m,kr to match those

obtained first for α’m,ks

• Simultaneous (Ben Akiva; Morikawa; Daly; Bradley)

Estimate α’m,kr using RP observations and α’m,k

r using SP observations and (λs/λr) altogether where (λs/λr) α’m,k

r is used in place of α’m,ks

• Little concensus on approach for βm

Joint RP/SP Data

• Used ALOGIT software

• Set β’m=1s = 0 and β’m=1

r = 0 to avoid over-specification

• Provides:• α’m,k

s = λs αm,k α’m,kr = λr αm,k

• β’ms = λs βm +βm

s β’mr = λr βm +βm

r

• λr/λs

• Know that λr = 0.855 and λs = 1.166

Joint RP/SP Ranking Estimation for Full set of RP and SP

15,000 Observations (7 Alternatives for each)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

observed

est

imate

d

Joint RP/SP Ranking Estimationwith 15,000 RP Observations for Alternative 1-4and 15,000 SP Observations for Alternatives 4-7

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

Observed

Est

imate

d

RP Ratings• Two potential interpretations of ratings

• Value provided is a (scaled?) direct utility• Value provided is 10x probability of

selection• Issue of reference

• ‘excellent’ in terms of other people’s travel• ‘excellent’ relative to other alternatives for

respondent specifically• Related to interpretation above

• Here: Use direct utility interpretation and thus reference is in terms of other people’s travel

RP Ratings• Used MINITAB MLE Provides:

• α’m,kg = θ

g αm,k

• β’mg = θ

g βm + βmg

RP Ratings Estimation with 15,000 Observations

-12

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

-12 -10 -8 -6 -4 -2 0 2 4 6 8 10 12

Observed

Estim

ated

Estimation of PlottedRP Ratings Values

• θ g is found by minimizing the

minimum square error between estimated sensitivities (θ

g αm,k ) and the true values αm,k

• The estimated values for βm are then found using (β’m

g - βmg

min)/ θ g with the above-determined value for θ

g

SP Ratings• Used MINITAB • Provides:

• α’m,kh = θ

h αm,k

• β’mh = θ

h βm + βmh

SP Ratings Estimation with 15,000 Observations

-12

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

-12 -10 -8 -6 -4 -2 0 2 4 6 8 10 12

Observed

Estim

ated

Estimation of PlottedSP Ratings Values

• θ h is found by minimizing the

minimum square error between estimated sensitivities (θ

h αm,k ) and the true values αm,k

• The estimated values for βm are then found using (β’m

h - βmh

min)/ θ h with the above-determined value for θ

h

Combined RP/SP Data and Ratings

• Purpose-built software• Log-Likelihood function: L = Σk Prob(ms*) + Σk Prob(mr*) - wg Σk Σm (Rm

gobs - Rm

gmod) 2

- wh Σk Σm (Rmhobs - Rm

hmod) 2

where:mr* = selected alternative in RP observationms* = selected alternative in SP observationProb(m) = probability model assigns to alternative m

Combined RP/SP Data and Ratings

Prob(ms*) = exp( [Σk α’m*,ks xm*,k] + β’m*

s)/ ( Σmexp( [Σk α’m,k

s xm,k] + β’ms ) )

Prob(mr*) = exp( [Σk α’m*,kr xm*,k] + β’m*

r)/ ( Σmexp( [Σk α’m,k

r xm,k] + β’mr ) )

Rmg

mod = Σk α’m,kg xm,k + β’m

g

Rmh

mod = Σk α’m,kh xm,k + β’m

h

Combined RP/SP Data and Ratings

• Consider range of results for βmr for

different settings on variables• Example planned settings

• Set θ h = 1

• Set wg and wh = 1

• This ‘anchors’ utilities to values provided in SP Ratings

Conclusions• Work in progress

• Not complete, but still discovering things

• βm estimators problematic generally• Even with existing alternatives • Not as efficient as those for αm,k

• Influenced by variation in attribute values σm,k

• Influenced by frequency of chosen alternatives?

• T-statistics not a useful guide?• Ranking (exploded) helps• Rating also expected to help