ProbabilisticPolyhedralMethodsfor AdaptiveChoice …hauser/Papers/Toubia_Hauser_Garcia... ·...

Vol. 26, No. 5, September–October 2007, pp. 596–610issn 0732-2399 eissn 1526-548X 07 2605 0596

informs ®

doi 10.1287/mksc.1060.0257©2007 INFORMS

Probabilistic Polyhedral Methods forAdaptive Choice-Based Conjoint Analysis:

Theory and Application

Olivier ToubiaColumbia Business School, Columbia University, 522 Uris Hall, 3022 Broadway, New York, New York 10027,

[email protected]

John HauserMIT Sloan School of Management, Massachusetts Institute of Technology, 40-179, 1 Amherst Street,

Cambridge, Massachusetts 02142, [email protected]

Rosanna GarciaCollege of Business Administration, Northeastern University, 202 HA, Boston, Massachusetts 02115, [email protected]

Polyhedral methods for choice-based conjoint analysis provide a means to adapt choice-based questionsat the individual-respondent level and provide an alternative means to estimate partworths when there

are relatively few questions per respondent, as in a Web-based questionnaire. However, these methods aredeterministic and are susceptible to the propagation of response errors. They also assume, implicitly, a uniformprior on the partworths. In this paper we provide a probabilistic interpretation of polyhedral methods andpropose improvements that incorporate response error and/or informative priors into individual-level questionselection and estimation.Monte Carlo simulations suggest that response-error modeling and informative priors improve polyhedral

question-selection methods in the domains where they were previously weak. A field experiment with over2,200 leading-edge wine consumers in the United States, Australia, and New Zealand suggests that the newquestion-selection methods show promise relative to existing methods.

Key words : conjoint analysis; choice models; estimation and other statistical techniques; internationalmarketing; marketing research; new-product research; product development; Bayesian methods

History : This paper was received May 26, 2006, and was with the authors 1 month for 1 revision; processed byEric Bradlow.

1. IntroductionToubia et al. (2003) demonstrated that polyhedralmethods for adaptively selecting questions in met-ric conjoint analysis could improve accuracy whenpartworths are either homogeneous or heterogeneous,and could do so whether response errors are large orsmall. Toubia et al. (THS 2004) extended polyhedralmethods to choice-based questions, but with mixedsuccess. Polyhedral choice-based questions improvedaccuracy when response errors were low, but notwhen they were high. Furthermore, although polyhe-dral methods for metric paired-comparison questionspredict well for empirical data, there have been noempirical validity tests for choice-based polyhedralmethods despite the growing interest among practi-tioners for adaptive choice-based methods.In this paper we propose and test a generalization of

THS that takes response error into account for choice-based questions and has the potential to improveaccuracy in high response-error domains. We do soby recasting the polyhedral heuristic into a Bayesian

framework. This framework also includes prior infor-mation in a natural, conjugate manner. After verify-ing the methods with simulations, we undertake alarge-scale, multicountry study in which each respon-dent completes two separate conjoint tasks. Thisdesign enables us to compare question selection witha within-respondent design that implies greater sta-tistical power to distinguish methods. We comparemethods on the ability to predict actual choices. Weexamine whether the methods lead to different man-agerial implications by comparing forecasts of will-ingness to pay as well as the optimal product linesimplied by each method.This paper is organized as follows. Section 2 briefly

reviews the published choice-based polyhedral meth-ods and discusses two key limitations. Sections 3 and4 propose solutions to these limitations. Sections 5examines the methods with Monte Carlo simulations.Section 6 describes the methodological results of thefield experiment. Section 7 concludes and offers direc-tions for future research.

596

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.

Toubia, Hauser, and Garcia: Probabilistic Polyhedral Methods for Adaptive Choice-Based Conjoint AnalysisMarketing Science 26(5), pp. 596–610, © 2007 INFORMS 597

2. Review and Critique of PolyhedralChoice-Based Methods

Choice-based polyhedral question selection selectseach choice question to learn as much as possibleabout a respondent’s preferences. The conceptual ideais to recognize that the set of choice questions andtheir corresponding answers define a polyhedron,i.e., a set of “feasible” partworth vectors that per-fectly fit previous observations. Each choice narrowsthe range of feasible partworths making the rangesmaller and smaller until it converges toward a sin-gle partworth vector. The method works well whenthe respondent makes no errors, but can be highlysensitive to errors, particularly in the early choices.We now provide a brief technical review to estab-lish both notation and conceptual reasoning for thegeneralizations.

Answers to Choice-Based Questions Interpreted asConstraints on the PartworthsWithout loss of generality, we use binary vectors inthe theoretical development to simplify notation andexposition. Multilevel features are used in both thesimulations and the application. Let xqjf indicate thatthe jth alternative in the qth choice set contains thef th feature, and let xqj be the binary row vectordescribing the jth alternative in the qth choice set.Define xqk similarly for the kth profile. Let u be thel-dimensional vector of partworths for a given respon-dent. Let qj and qk be error terms such that therespondent’s utility for profile j in choice set q is xqj u+ qj . The utility-maximizing respondent will chooseprofile j∗ over profile k if and only if xqj∗ − xqku+ qj∗ − qk ≥ 0. Each choice among J alternativesimplies J − 1 such inequality constraints, indicatingthat the utility of the chosen profile is higher thanthat of the other J − 1 alternatives in the choice set.Let X 1q be the matrix of the xqj∗ − xqks for allJ −1 inequality constraints stacked for the first q ques-tions. Note that the respondent’s q choices are codedin X 1q by the selection of j∗ for each question.Let be the corresponding vector of error differencesand, without loss of generality, scale all partworths tobe nonnegative and normalize the partworths so thatthey sum to 100.1 Then, if e is a vector of 1s and 0 isa vector of 0s (of length l, the answers to the choice-based questions imply the following constraints onthe respondent’s partworths:

P1 X 1q u+ ≥ 0 u≥ 0 e′ u= 100

1 Nonnegativity assumes that we know a priori which level of thepartworth vector is preferred. This simplifies notation. We addressempirical issues in later sections. The selection of 100 is arbitraryand implicitly rescales the error vector, .

Figure 1 Deterministic Polyhedral Question Selection

Polyhedron Ω1,…,q = set of estimates

Longest axis ofthe polyhedron

Set of estimatesconsistent withx(q + 1)1 being chosen

Set of estimatesconsistent withx(q + 1)2 being

Analytic center

chosen.

consistent with the first q choices

Question SelectionFor any given vector , the set of vectors u satisfy-ing the constraints in (P1) is a mathematical objectcalled a polyhedron. THS select questions such thatthe polyhedron corresponding to = 0 never becomesempty, and effectively assume that = 0. Let 1qbe the polyhedron obtained after q questions. Theq+ 1st question imposes new constraints on the part-worths, giving rise to a new polyhedron 1q+1 thatis a subset of the previous polyhedron 1q. Fora linear compensatory utility model, each point in 1q is consistent with the respondent choosing oneand only one of the alternatives in choice set (q + 1)(except for a set of points of measure 0 for which therespondent is indifferent between at least two pro-files). Hence, the q+ 1st question divides 1q intoJ collectively exhaustive (smaller) polyhedra that areof roughly equal size. The region corresponding to therespondent’s choice becomes the starting polyhedronfor the next question. See Figure 1 for a choice setof two alternatives. If there were no response errors,the sequence of polyhedra would shrink toward therespondent’s true partworth vector.Question selection (choice set selection) obeys two

principles: (1) choice balance and (2) postchoice sym-metry. Choice balance minimizes the expected sizeof 1q+1 and is implemented by ensuring that arespondent who uses the working estimate of thepartworths, uq , would be approximately indiffer-ent between all the alternatives in the choice set.Choice balance is common in the literature and, forchoice questions, typically increases the efficiency ofthe questions (Arora and Huber 2001, Hauser andToubia 2005, Huber and Zwerina 1996, Kanninen2002).2 Postchoice symmetry minimizes the maximum

2 The first-order conditions for logit-based choice-based questionsindicate that the information matrix is maximized for questionsthat are close to, but not perfectly, choice balanced. See appendixto Hauser and Toubia (2005).

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.

Toubia, Hauser, and Garcia: Probabilistic Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis598 Marketing Science 26(5), pp. 596–610, © 2007 INFORMS

uncertainty on any combination of partworths, and isimplemented by constructing choice sets that dividethe polyhedron 1q perpendicularly to its longestaxes.

EstimationBecause choice questions are chosen such that thepolyhedron 1q never becomes empty, all pointsin 1q are consistent with all of the respondent’schoices. Thus, THS use the analytic center of 1q,uq , as the working estimate of u after q questions.

CritiqueChoice-based polyhedral question selection and esti-mation are promising. Empirically, choice balance isachieved and the polyhedra shrink rapidly (althoughthere is no published data on the ability to pre-dict actual choices). Compared to randomly gener-ated questions, orthogonal designs, and aggregatecustomization (Arora and Huber 2001, Huber andZwerina 1996), deterministic choice-based polyhedralquestions improve accuracy when response error islow, but not when response error is high.The poor performance for high response errors is

likely due to response-error propagation, as illus-trated in Figure 2. In this example, the respondent’strue partworth values are as indicated by a star ().With no response error, the respondent would chooseProfile 2, corresponding to the lower polyhedron,and the set of feasible partworths (new polyhedron)would converge toward the true value. However,with response errors the respondent might chooseProfile 1, corresponding to the upper polyhedron.Once such a choice is made, the partworths can neverconverge to the true value. The closest estimate wouldbe on the border, as indicated by the small dia-mond (). Moreover, without a formal probabilisticstructure, there is no easy way to incorporate prior

Figure 2 Illustration of Response Error in Deterministic PolyhedralQuestion Selection

Polyhedron Ω1,…,q = set of estimates

Set of estimatesconsistent withx(q + 1)1 being chosen

Set of estimatesconsistent withx(q + 1)2 being

Estimates caughton the “wrong” sideof the polyhedrondue to response error.

True partworthschosen.

consistent with the first q choices

information on the likely distribution of partworths.We next address both response error and prior infor-mation with a Bayesian interpretation of choice-basedpolyhedral methods.

3. Bayesian Interpretation forChoice-Based Polyhedral Methods

We can interpret the analytic center as a working esti-mate if we assume a prior distribution on the part-worth vector u that is uniformly distributed on theinitial polyhedron, 0 = u u ≥ 0 e′ u = 100. Denotethis distribution as P0

: defined by P0 u= 0 if u0;

P0 u = 0−1 if u ∈ 0. (0 is the measure of

the set 0.) Denote the data provided by the respon-dent through the first q choices with D 1q. (D 1qis encoded in X 1q.) The deterministic algorithmimplicitly assumes a likelihood function of the form:PD 1q u = 1 if u ∈ 1q and PD 1q u = 0if u 1q. Applying Bayes rule, P u D 1q ∝PD 1q uP0

u ∝ P 1q u. In other words, the

posterior distribution is the uniform distribution withsupport 1q.Once the method is viewed in a Bayesian frame-

work, the two implicit assumptions of the absenceof response error and uniform priors may easily berelaxed by generalizing, respectively, the likelihoodfunction and the prior.

4. Probabilistic Polyhedral MethodsGeneralizing the Likelihood FunctionIn the deterministic algorithm = 0, and the respon-dent chooses the profile with the highest deterministicutility with probability 1. All posterior distributionsare uniform distributions supported by polyhedra.We generalize the algorithm by considering distri-butions supported by mixtures of polyhedra. As thenumber of polyhedra in the mixtures grows, we canapproximate any distribution, but we must balancethis capability with the realization that as the numberof polyhedra grows, the computational time growsexponentially. To balance these effects, we choose asimple likelihood function that captures the essenceof response error. We use simulations to examinewhether this is a sufficient approximation.To obtain a structure in which the prior and pos-

terior distributions are conjugate, we assume thatthe noise is distributed such that the respondentchooses the profile with the highest deterministic util-ity with probability, ′, and chooses the J − 1 otherprofiles with probability 1− ′/J − 1. The advan-tages of this assumption are that it provides a feasi-ble algorithm and nests the deterministic algorithmas the special case when ′ = 1. While we believethis assumption is a reasonable, first-order robust

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


assumption, it may not hold exactly in real or syn-thetic data. To test the robustness of this assump-tion, we generate data in our simulations that use atraditional logistic function and, hence, violate thisassumption to some degree.In general, ′ is unknown and can be assumed

to vary across respondents and, potentially, acrosschoice sets within a respondent (e.g., ′ may behigher or lower if the profiles in the choice set arecloser in utility). We might include priors for ′

and do a full Bayesian updating such that P u ′i

D 1q xi1xi2 xiJ ∝ ∏

i P Di u′iP ′

i uxi1xi2 xiJ P u.To avoid complexity, for a first test of the algo-

rithm, we model ′ as homogeneous and constant.Fortunately, sensitivity analyses suggest that predic-tive ability is not sensitive to the choice of ′ withina wide range that is consistent with the ′s estimatedfor our simulations and empirical test. See Appen-dices A1 and A2 for details on estimation and sensi-tivity. We leave to future research the investigation ofalternative ways to specify and estimate ′. For ourempirical tests, we use pretest data to select a pointestimate of ′. Pretest selection follows the traditionof aggregate customization (Arora and Huber 2001,Huber and Zwerina 1996).

Generalizing the Prior DistributionWe nest THS’s implicit prior distribution within amixture of uniform distributions supported by poly-hedra:

∑Mm=1mPm

u where M is any positive inte-ger, 1 M is a set of positive weights such that∑M

m=1m = 1, and 1 M is a set of polyhedra.In this paper we apply and test two special cases

of nonuniform priors. The first special case approx-imates traditional normal priors. Figure 3 illustratesthe approximation conceptually. (In one dimension,a polyhedron is a line segment.) The uniform distribu-tions are indicated with solid lines; the approximationwith a dotted line. Appendix A3 provides a proce-dure for choosing weights for the polyhedra provid-ing support for the distribution.The second special case, denoted “population pri-

ors,” selects a mixture of polyhedra such that themedian of the prior importance of each feature isequal to the median (across respondents) of its impor-tance. The polyhedra are defined by inequalitiesimplied by the median importances of the features.If F is the number of features, this prior uses a mix-ture of 2F polyhedra.3 Details on the definitions ofthe polyhedra and the computation of the weights aregiven in Appendix 4.

3 The importance can be defined as the difference between themaximum and minimum partworths for a feature or the aver-age absolute magnitudes of the partworths. Importances have bothconceptual and computational advantages relative to imposingmedian constraints on the partworths directly.

Figure 3 Approximating a Normal Prior with a Mixture of Polyhedra

Conjugate Posterior DistributionsAn important feature of our generalization is that theclass of likelihood functions and the class of priorspresented above are conjugate; that is, the posteriordistributions remain within the set of mixtures of uni-form distributions supported by polyhedra.In order to show this result, we begin with a prior

distribution, P0 u, supported by a (single) polyhe-

dron, 0. Let 1 = u X1 u≥ 0 u≥ 0 e′ u= 100 be thepolyhedron defined by the answer to the first ques-tion and D1 be the data provided by this question.Let 0 −1 be the set in which all points in 1 areremoved from 0. Applying Bayes rule:

p u D1 ∝ pD1 uP0 u

=

′0−1 if u ∈1(1−′

J − 1)0−1 if u ∈0−1

(1)

The posterior is proportional to a piecewise-constantfunction that takes the values of zero at all points out-side 0, ′0−1 at all points in 1, and 1−′/J − 10−1 at all points in 0 −1. Hence, thereexists a scalar, ∈ 01, such that:

p u D1 = P1 u+ 1−P0

u

=1−1+ 1−0−1 if u ∈1

1−0−1 if u ∈0−1(2)

where P1 u is the uniform distribution with sup-

port 1. Equation (2) demonstrates that the posterioris a mixture of two uniform distributions supportedby polyhedra. The scalar is implicitly defined byequating Equations (1) and (2):

1−1+ 1−0−11−0−1

= ′J − 11−′

⇔ 0/1+ 1−

1−= ′J − 1

1−′

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


The computation of requires knowledge of0/1, which is the ratio of the measure of 0to the measure of 1. For the choice-based polyhe-dral algorithm, we seek choice balance such that 0 isdivided into J collectively exhaustive equal-measuresubpolyhedra; thus, the ratio 0/1 is close to itsaverage J . Hence, J′ − 1/J − 1.Let us next consider prior distributions that are

defined by any mixture of uniform distributions sup-ported by polyhedra,

∑Mm=1mPm

u. Defining 1 asabove and following the same argument, the posteriorafter the new choice question is proportional to:

PD1 u[ M∑m=1

mPm u

]

=M∑

m=1mPD1 uPm

u

=M∑

m=1mmPm∩1

u+ 1−mPm∩0 u

which is also a mixture of uniform distributions sup-ported by polyhedra.Finally, we generalize to q questions. Let Sq be the

set of all subsets of 12 q. For a subset s

of Sq , let s = u Xs u ≥ 0 u ≥ 0 e′ u = 100 be thepolyhedron consistent with the choice questions con-tained in s (recall that Xs encodes the constraintsimplied by the answers to the questions in s). Letws = s1 − q−s for all nonempty s , where sdenotes the number of elements in subset s and =J′ − 1/J − 1. The posterior after q questions is amixture of uniform distributions supported by thepolyhedra s ∩ms∈Sm∈ 1M. We approximate theweights as follows:4

Pq u=M∑

m=1

∑s∈Sq

mwsPs∩m u (3)

We denote question selection and estimation basedon this posterior distribution as “polyhedral witherror-modeling and informative priors.” We also con-sider the following special cases in the simulationsand field experiment:• “polyhedral without error-modeling and with

uniform priors” (as in THS): = 1, prior: P0 u, pos-

terior: Pq u= P 1q u;

• “polyhedral with error-modeling and with uni-form priors:” < 1, prior: P0

u, posterior: Pq u =∑s∈Sq wsPs

u;• “polyhedral without error modeling and with

informative prior:” = 1, prior: ∑Mm=1mPm

u, pos-terior: Pq u=

∑Mm=1mP 1q∩m

u.

4 We set mws to zero if s ∩m = and normalize the weights tosum to one.

Selecting Questions and Estimating Partworthswith Mixtures of DistributionsIn the deterministic algorithm, THS select questionsbased on the analytic center and longest axes ofa single polyhedron. This is a well-defined prob-lem. For the probabilistic algorithm we must workwith Pq u, which is a mixture of uniform distribu-tions supported by polyhedra. To implement choicebalance and postchoice symmetry, we must computethe analytic center and longest axes of polyhedralmixtures. Fortunately, the analytic center may simplybe replaced with the appropriate mixture of the ana-lytic centers of the polyhedra in the mixture. How-ever, computing the longest axes poses a conceptualand technical challenge.

Longest Axes of a Mixture of PolyhedraThe longest axis of a mixture of polyhedra shouldsummarize the directions of the longest axes of thepolyhedra in the mixture and do so according to theweights of the mixture. Let vsm be the longest axis ofthe polyhedron s ∩m (see Equation (3)), and mws

be the corresponding weight. We seek the vector v∗

that maximizes the weighted norm of the projectionsof vsm on v∗. Thus, the longest axis is the solution tothe following mathematical program:

OPT1 v∗q = argmax

v

M∑m=1

∑s∈Sq

mwsvTsm v2

Fortunately, OPT1 has a known solution. Define Vas the matrix obtained by stacking the transposedlongest axes, vT

sm. Define ) as the diagonal matrixwith elements equal to the weights mws. We rewriteOPT1 in matrix form as follows:

M∑m=1

∑s∈Sq

mwsvTsm v2 = v′V ′)V v

OPT1 is now a standard optimization problem that isanalogous to factor analysis: v∗

q is the eigenvectorassociated with the largest eigenvalue of V ′)V . Thematrix is symmetric and positive definite; hence, itseigenvalues are all real and nonnegative. The second-longest axis is associated with the second eigenvalue,etc. Because the axes are eigenvectors, they are guar-anteed to be orthogonal.

Practical ImplementationWhile mixtures of polyhedra can approximate almostany distribution, there are practical considerations.Not only do the population priors grow exponentiallywith the number of features (2F ), but the number ofsubsets Sq in Equation (3) grows exponentially withthe number of questions (2q). For small q and small F ,computation can be done quickly. Choice-based ques-tions can be selected in less than a second, such that

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


respondents do not notice any delay. However, forlarge q or large F the delay can exceed a second (e.g.,216 = 65536).We take three steps to reduce computation time.

First, the set of polyhedra in the posterior mixtureafter q questions, Ps∩m

m=1M* s∈Sq , is a subset ofthe polyhedra in the posterior mixture after q + 1questions, Ps∩m

m=1M* s∈Sq+1 (this follows from thefact that Sq ⊂ Sq+1. By saving, rather than recomput-ing, the longest axes and analytic center, we reducecomputation time substantially. Second, one of thetime-consuming steps in polyhedral methods is find-ing a feasible point in s ∩m. If a point is feasiblein 12q ∩ m, then it is feasible for all s ∩ m,s ∈ Sq . By reusing feasible points, we also reduce com-putational time substantially. Third, as the numberof questions grows large, we sort the weights mws

in decreasing order and apply the algorithm to sub-sets corresponding to the largest weights, doing sountil a preset time limit is reached. In our empiricalwork, that time limit is one second. In simulation, weuse 10 seconds. Exploratory work suggests that thesetime limits provide excellent performance. However,all empirical and simulation results reported in thispaper can be considered conservative and mightimprove slightly with faster computers and more effi-cient codes/programming/compilers.

5. Monte Carlo SimulationsModeling response error and informative priorspromises to enhance the accuracy of choice-basedpolyhedral question selection. However, both exten-sions increase complexity and could result in over-fitting the data. To evaluate performance, we turnto complementary testing tools, both synthetic andempirical data. We use Monte Carlo simulations tostudy the potential of the methods by investigatingthe range of performance in a variety of relevantdomains. With synthetic data we know the “truth”and can compare estimates to that benchmark. We usethe field experiments to test practical implementationin a realistic setting. We do not know the true valuesof the partworths, and so must use predictive abilityas a surrogate.

Experimental Design for the Monte CarloSimulationsWe use a 2×2×6×4 design for the Monte Carlo simu-lations. We simulate the respondents with a 2×2 sub-design that is becoming standard—allowing for twolevels of response accuracy and two levels of respon-dent heterogeneity (Arora and Huber 2001, Evge-niou et al. 2005, Toubia et al. 2004). The Arora-Huberdesign, as modified by THS, uses four features at fourlevels each to ensure complete aggregate customiza-tion and orthogonal designs. Partworths are drawn

from normal distributions with means, u, and vari-ances, +2

,. The four levels of each partworth havemeans [−,−,/3 ,/3 ,]. Higher values of , implyhigher response accuracy. Higher values of +2

, implygreater heterogeneity. We used the standard valuesof , = 1 for the “low-accuracy” case and , = 3 forthe “high-accuracy” case with +2

, = , for “low hetero-geneity” and +2

, = 3, for “high heterogeneity.”For each respondent we simulate six question-

selection methods:• random• orthogonal• aggregate customization (Huber and Zwerina

1996, Arora and Huber 2001)• deterministic polyhedral (as in THS)• probabilistic polyhedral with error modeling and

uniform priors• probabilistic polyhedral with error modeling

and informative prior (prior approximates a normaldistribution—see §3 and Appendix A3 for details).The six question-selection methods are crossed with

four estimation methods:• hierarchical Bayes (HB) with normal priors• deterministic analytic-center estimation (AC)• analytic-center estimation with error modeling

and uniform prior (ACe)• analytic-center estimation with error modeling

and informative prior (ACe+i) (prior approximates anormal distribution).

Simulated EnvironmentAggregate customization uses relabeling and swap-ping to improve utility balance in choice-based ques-tions and requires an estimate of the populationpartworth means. Polyhedral question selection witherror modeling requires an estimate of ′. This esti-mate is derived from the same population estimates(see Appendix A1 for details). Following Arora andHuber (2001), we assume perfect pretest informa-tion. This should not affect the relative comparison ofaggregate customization and probabilistic polyhedralquestion selection. Naturally, no such assumption ismade in the empirical tests. Likewise, to investigatethe impact of informative priors (relative to no pri-ors), we use a rough approximation (four polyhedra)to the true prior distribution.5 All simulation resultsare interpreted in light of these assumptions.We seek to afford the estimation benchmark meth-

ods the strongest possible performance. Evgeniou

5 We use normal priors rather than population priors on featureimportances, because the latter would require that we deviate fromthe standard simulation design, thus reducing our ability to com-pare our results to previously published papers. Population priorsonly lead to differences when the average importances vary amongthe four simulated features. This does not happen in the standardsimulation design, but is likely in our empirical test.

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


Table 1 Monte Carlo Simulation Results

RMSEMagnitude(accuracy) Heterogeneity Question selection method HB1 AC ACe ACe+i

Low High Random 0669 0932 0808 0728Orthogonal 0644 0902 0705 0658Aggregate customization 0610 0916 0733 0670Deterministic polyhedral 0599 0778 0648∗ 0614Probabilistic polyhedral w/ error 0586∗ 0759∗ 0645∗ 0601∗

modeling and uniform priorsProbabilistic polyhedral w/ error 0584∗ 0767∗ 0645∗ 0598∗

modeling and informative priors

Low Low Random 0645 0963 0824 0653Orthogonal 0604∗ 0913 0714∗ 0606Aggregate customization 0597∗ 0983 0786 0627Deterministic polyhedral 0630 0877 0708∗ 0607Probabilistic polyhedral w/ error 0612 0837∗ 0713∗ 0603∗



High High Random 0583 0887 0850 0662Orthogonal 0562 0939 0729 0586Aggregate customization 0514 1026 0785 0613Deterministic polyhedral 0497 0680 0613∗ 0528Probabilistic polyhedral w/ error 0487 0665 0620∗ 0521



High Low Random 0489 0903 0861 0580Orthogonal 0450 0959 0746 0474Aggregate customization 0404 1057 0815 0499Deterministic polyhedral 0441 0702 0623∗ 0468Probabilistic polyhedral w/ error 0438 0677∗ 0633∗ 0480

modeling and uniform priorsProbabilistic polyhedral w/ error 0392∗ 0671∗ 0648 0422∗


Notes. RMSE, lower is better.∗Best, or not significantly different from best, at p≤ 005 within magnitude× heterogeneity× estimation condition.1HB= hierarchical Bayes estimation, AC= deterministic analytic-center estimation, ACe= analytic-center estimation with error modeling

and uniform priors, ACe+i= analytic-center estimation with error modeling and informative priors.

et al. (2005) demonstrate that HB performs better if foreach respondent we use rejection sampling (Allenbyet al. 1995) to constrain the HB estimates so that thepartworth of the lowest level of each feature is alsothe smallest partworth for that feature.6 We adopt thisprocedure for HB in both the simulation and the fieldexperiments.Because polyhedral methods are designed for short

Web-based questionnaires, we test designs of eightquestions, choosing randomly for orthogonal andaggregate customization as in THS. For comparisonto previously published simulations we report rootmean squared error (RMSE) after normalizing the trueand the estimated partworths so that their absolutevalues sum to the number of parameters and so thattheir values sum to zero for each feature. This enables

6 Constraints in estimation are used in other areas of marketing aswell (see, for example, Ailawadi et al. 2005).

us to interpret the RMSEs as a percentage of the mean(absolute) partworths. The simulations in Table 1 arebased on now-standard 10 sets of 100 respondents.This is not a computational constraint; the field testsare based on larger samples.

Results and Interpretation ofSynthetic Data Experiments

Question Selection. Taking response errors intoaccount and using informative priors appears to havethe potential to improve question selection. At leastone of the two modifications in polyhedral questionselection is best or tied for best in all 16 accuracy×heterogeneity× estimation experimental cells. Proba-bilistic polyhedral question selection with error mod-eling and uniform priors is at least as good as thedeterministic algorithm in every cell and significantlybetter in 9 of the 16 cells. Probabilistic polyhedral

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


question selection with error modeling and informa-tive priors is at least as good as the deterministicalgorithm in 15 of the 16 cells and significantly bet-ter in 12 of the 16 cells. The field experiment will testwhether such improvements are sustained in practicalimplementations.

Estimation. Taking response errors into accountand using informative priors also appear to have thepotential to improve polyhedral estimation. At leastone of the two improvements is better than deter-ministic analytic-center estimation in all accuracy ×heterogeneity experimental cells. Informative priorsappear to provide the greater improvement. How-ever, the hierarchical Bayes estimates (HB) are stillsignificantly better in three of the four accuracy ×heterogeneity cells. The only exception is the low-accuracy, low-heterogeneity cell in which ACe+i isstatistically tied with HB. These results are consistentwith the simulations of Evgeniou et al. (2005).In summary, our simulations suggest that incorpo-

rating response error and/or informative priors intopolyhedral question selection is likely to enhance ac-curacy in empirical applications. Analytic-center es-timation is also improved, but hierarchical Bayes islikely to remain the best estimation method in mostapplication domains for choice-based questions.

6. Empirical Application and TestManagerial ContextTraditional cork closures have dominated the wineindustry for hundreds of years, but each year 5%–15%of all bottled wine is tainted due to poor-qualityclosures. Cork closures result in brand-name erosionand millions of dollars in lost revenue when con-sumers attribute the poor quality to the winery ratherthan the closure. As an alternative to cork closures,the wine industry developed Stelvins, a screw-cap/twist-off closure for mid-to-high-priced wines.Stelvins eliminate cork taint and other malodorousflavors, eliminate wine oxidation that leads to rapidaging, and minimize loss of fruit flavors due to airleakage. Stelvins provide “consistent, reliable, agingcharacteristics, showing the wine’s development asthe winemaker intended (Courtney 2001).”Although Stelvins have been available for almost

50 years, and in Australia and New Zealand salesof premium wines with Stelvins now outnumber thesales of premium wines with corks, Stelvins are rarelyused in the United States. To explore strategies fora U.S. introduction of Stelvins, a Napa Valley-basedclosure manufacturer and cooperating U.S. wineriesasked us to determine preferences of leading-edgewine customers in the United States, New Zealand,and Australia.

Experimental DesignIn exchange for gathering these data, the sponsorsagreed to set up the application as an experimentaldesign. Each respondent completed two sequential,rotated, choice-based conjoint analysis tasks separatedby a series of “memory-cleansing” questions. The ad-vantage of this experimental design is the increasedpower due to methodological comparisons withinrespondents.We recruited 2,255 leading-edge wine consumers

from the United States, Australia, and New Zealand(late 2004). Respondents were subscribers to wine-related e-newsletters (WineX and WineBrats in theUnited States, Vine Cellars in Australia and NewZealand) and could be expected to be knowledge-able about fine wines. They were likely to be leading-edge consumers. As a check, 80% of the respondentsscored 15 or higher on a 21-point involvement scale(Lockshin et al. 2001). We obtained 245 respondentsfrom a first U.S. panel, 958 from a second U.S. panel,667 from Australia, and 385 from New Zealand. Asis typical in managerial applications, we did not havetotal control over the assignment of respondents totreatments, although there was no reason to believethat there were any systematic biases within any ofthe countries.Managerially, the sponsors were interested in the

trade-offs that the consumers would make betweenwine closures and other features of wine. The conjointdesign included five features at four levels each:• closure type: traditional cork, synthetic cork,

Metacork™,7 Stelvin (screw cap);• type of wine: dry white, aromatic white, dry red,

blush red;• origin: Australia/New Zealand, France, Sonoma/

Napa, Chile/Argentina;• vintner: small boutique, midsize regionally

known winery, large nationally recognized winery,international conglomerate winery;• price range:8 four levels in the respondents’ cur-

rency (e.g., Australian dollars).The features in the conjoint design were intro-

duced to respondents through self-explicated impor-tance questions.9 Figure 4a shows the closure types

7 A MetaCork™ “combines an integrated corkscrew, a drip-resistantpour feature, and a reseal cap.” (www.metacork.com, viewed 2006)8 Based on pretests, respondents felt they could best evaluate thechoices among wines if price was specified as a range. This is suffi-cient for relative methodological comparisons and the study of theimpact of consumer preferences for Stelvin closures.9 These answers allowed us to identify the lowest level of eachfeature. This information was used in adaptive question selectionand by all the estimation methods, including HB (see previous sec-tion). The self-explicated information was used in adaptive ques-tion selection and in estimation in order to avoid endogeneityand/or violations of the likelihood principle (Liu et al. 2006).

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


Figure 4 Example Screenshots from the Wine-Closure Preference Study

(a) Wine closures (b) Another feature (winery type)

(c) Choice-based questions (d) Validation choice questions

Wine Purchasing Choices Wine Purchasing Choices

Choose a Prizeto possibly winChoose a

Wine for Everyday Drinking at Homewith Family or Close Friends

Next Next0 25 50 75 100 0 25 50 75 100

0 25 50 75 100

0 25 50 75 100

All else being equal, I prefer to buy wines with the followingclosure types: (If you are unfamiliar with a closure type, click onthe photo to link to an explanation or demonstration of the closure)

All else being equal, I prefer to buy wines from thefollowing type of wineries:

Traditional Cork

Small boutique wineries with limited production(for example, less than 5000 cases annually)

Mid-sized regionally known wineries(for example, less than 100,000 cases annual production)

Large nationally recognized wineries(for example, less than 1 million cases annual production)

International conglomerate wineries(for example, over 1 million cases annually)

Synthetic Cork

From the choices presented here, please select your most preferred choice.Question 1 of 12 for this section

Screwcap (also called Stelvins)

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

StronglyDisagree

StronglyAgree

Neutral

MetacorkTM

1 2 3 4 5 6 7

1 2 3 4 5 6 7

1 2 3 4 5 6 7

1 2 3 4 5 6 71 2 3 4 5 6 7

1 2 3 4 5 6 7

1 2 3 4 5 6 7

1 2 3 4 5 6 7

FeaturesChoice A Choice B Choice C Choice D

Features Choice A Choice B Choice C

WineType

ClosureType

Region

PriceRange

Typeof

Winery

Aromatic White Aromatic White Aromatic White

Australia/NZ Australia/NZ

Aromatic White

Sonoma/NapaCalifornia USA

Traditional Cork

Small Boutique Small Boutique Small Boutique Mid-Sizedregionally known

Small Boutique Small Boutique

Traditional Cork Traditional CorkMetacork

Metacork Metacork

S. America (Chile,Argentina)

$AU15.00-$19.99 $AU15.00-$19.99 $AU15.00-$19.99 $AU15.00-$19.99

WineType

ClosureType

Region

PriceRange

Typeof

Winery

WineType

ClosureType

Region

PriceRange

Typeof

Winery

Aromatic White Blush Red

Aromatic White

Aromatic White

Australia/NZ Sonoma/NapaCalifornia USA

Sonoma/NapaCalifornia USA

Synthetic Cork

Mid-Sizedregionally known

Large nationallyrecognized

Internationallyrecognized

Internationallyrecognized

Traditional Cork

Traditional Cork

Traditional Cork



$AU20.00-$29.99

$AU20.00-$29.99 $AU15.00-$19.99 $AU15.00-$19.99

$AU20.00-$29.99 $AU30.00+

We will enter you into a drawing in which we will select a winner to receive a wineclub package worth $100 of wine. Please select your preferences for the followingwine selection should you be chosen as the winner.

As there might be limited supplies of the different available wines, please rankyour preferences for the following selections from first to sixth (where a first choiceindicates your highest preference and a sixth choice is your lowest preferences).

Second Fourth Fifth

Features Choice D Choice E Choice FSixth Third First

Dry RedDry Red

France

and Figure 4b another feature (vintners). Respondentswere then asked two sets of 12 choice-based ques-tions as illustrated by Figure 4c. The first 10 questionsof each set were designed by a different method (theorder was rotated). The last two questions were ran-domly selected holdouts. Finally, after additional filler

tasks, respondents were entered into a lottery with a 1in 200 chance of winning a case of wine worth $100.Respondents were asked to rank six cases of wine andwere told that they would receive their first choice ifit was available. Otherwise, they would receive theirsecond choice, etc. All bottles within a case were the

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


same (for example, if the wine costs $20, they receivedfive bottles).10 The six wine profiles were randomlychosen from a 16-profile orthogonal design. This lasttask, designed with a different look and feel from theconjoint tasks (Figure 4d), serves as a validation.We note that this study is the first empirical test

of the predictive ability of choice-based polyhedralmethods. Toubia et al. (2003) report on metric pairedcomparisons for laptop computer bags, and THSreport on convergence and choice balance for an exec-utive education study.

Comparisons of Question-Selection Methods:Experimental DesignWe tested the following four question-selectionmethods:11

• orthogonal design;• aggregate customization;• deterministic polyhedral (THS);• polyhedral with error modeling and uniform pri-

ors (“probabilistic polyhedral”).We chose the four methods carefully both to test the

new probabilistic polyhedral method and to exploretwo fundamental characteristics. (1) Adaptation: Bothdeterministic and probabilistic polyhedral methodsadapt questions within the respondent; aggregate cus-tomization and orthogonal designs do not. (2) Pretestinformation: aggregate customization and probabilis-tic polyhedral methods require pretest informationto set “tuning” parameters; orthogonal designs anddeterministic polyhedral methods do not. The pretestinformation was obtained from an HB analysis of66 respondents who answered questions based onan orthogonal design. In order not to confoundthese two characteristics, we tested these methodsin two pairs: orthogonal versus deterministic polyhe-dral, and aggregate customization versus probabilisticpolyhedral.12 We did not test random question selec-tion because prior research suggests that aggregate

10 Due to legal issues regarding alcohol as a prize, Australianrespondents were not eligible to win real cases of wine. For theserespondents the choice was hypothetical. Providing a reward witha fixed monetary value mitigates any wealth effect that might bepresent if we had endowed each respondent with money and giventhem the option of choosing among differently priced wines. Thetask remains incentive compatible as long as consumer utility isapproximately linear in the number of bottles of wine over therange of the options available.11 The orthogonal and aggregate customization designs were themost D-efficient sets of 10 questions from a 16-question orthog-onal design and from the corresponding aggregate customizationdesign, respectively.12 Due to a programming error, 227 and 204 respondents from theAustralian panel were assigned, respectively, to orthogonal ver-sus aggregate customization and to deterministic polyhedral versusprobabilistic polyhedral. Orthogonal versus aggregate customiza-tion revealed no difference. Probabilistic polyhedral performed bet-ter than deterministic polyhedral, although the sample size was

customization and orthogonal design are strongerbenchmarks. We leave tests of informative priors inquestion selection to future research. We test informa-tive priors for estimation (see below).

Comparisons of Estimation MethodsEach of the estimation methods is compatible with allof the question-selection methods enabling us to makecomparisons within respondent. The methods wetested were:13

• hierarchical Bayes estimation (HB);• deterministic analytic-center estimation (AC);• analytic-center estimation without error model-

ing and with informative population priors (ACi);• analytic-center estimation with error modeling

and with uniform priors (ACe).Because partworth values might vary by panel and

treatment (and some methods shrink estimates to themean or median), we apply all methods within paneland treatment.We begin with estimation and then move to our

primary focus: question selection. Table 2 summarizesthe comparisons of estimation methods for the valida-tion task by reporting the correlation (averaged acrossrespondents and question selection methods) betweenthe predicted and observed rankings of the six winesin the validation task.14

We compared estimation methods statistically witha repeated-measures ANOVA, with performance asthe dependent variable, two between-subject factors,panel (four levels) and question-selection comparisontreatment (two levels); and two within-subject factors,estimation method (four levels) and a factor captur-ing whether the question selection is adaptive (twolevels). We used contrast analysis to compare estima-tion methods. As predicted by the simulations, HBperforms significantly better than the other estimationmethods (p < 001).15

insufficient to reach significance. Details are available from theauthors.13 As is appropriate for an empirical test, we use our second formof informative priors in which the prior distributions are chosen tomatch the population medians. This method uses a mixture of 2F

polyhedra in the prior, where F = 5 is the number of features. Evenwith the computational efficiencies discussed earlier, it is not yetpractical to apply both error modeling and population priors on ourlarge data set. The former is exponential in the number of questionsand the latter in the number of features. We leave developmentof faster heuristics to future research, noting that empirical resultsare thus conservative. If error modeling and population priors areseparately promising, we might infer that their combination is alsopromising.14 Measuring performance by the proportion of respondents forwhom the first choice in the validation task was correctly predictedor using holdout hit rate yields similar qualitative implications.15 The panel was significant at the 0.01 level and treatment wasnonsignificant (p= 042). Adaptation is discussed below.

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


Table 2 Comparing Estimation Methods—Correlation with Choice

Australian panel New Zealand panel First U.S. panel Second U.S. panel AverageExperimental cell (n= 667) (n= 385) (n= 245) (n= 958) (n= 2255)

Hierarchical Bayes (HB) 0512 0421 0408 0337 0411Analytic center w/o error modeling 0453 0426 0365 0243 0350and w/ uniform prior (AC)

Analytic center w/ error modeling 0465 0420 0353 0279 0366and w/ uniform priors (ACe)

Analytic center w/o error modeling 0475 0434 0389 0245 0361and w/ informative priors (ACi)

Although the focus of this paper is on prob-abilistic polyhedral question selection, probabilisticanalytic-center estimation is a byproduct of proba-bilistic question selection, and probabilistic analytic-center estimation improves predictions relative todeterministic analytic-center estimation. Includingpopulation priors (ACi) significantly improves perfor-mance (p < 001) compared to deterministic analytic-center estimation (AC). Including error modeling(ACe) improves performance as well, albeit not sig-nificantly (p= 030). (ACe is never significantly worsethan AC, and is significantly better on the second U.S.panel.) The two improvements do not perform signif-icantly differently (p= 018).Empirical Comparison of Question-SelectionMethodsTable 3 reports the average performance of the dif-ferent question-selection methods (averaged acrossrespondents and estimation methods). Due to cir-cumstances beyond our control, all U.S. respondentswere assigned to the “probabilistic polyhedral versusaggregate customization” condition. Notice that theaverage predictive ability varies between panels, withpredictive ability significantly lower in the U.S. pan-els than in the Australian or New Zealand panels.While tempting, we cannot attribute these differencesto across-country variation. Our panels were chosenfrom opt-in organizations of leading-edge wine users.These organizations might vary on other characteris-tics besides country of origin. Nonetheless, a futureinvestigation of across-country differences in responsequality would be interesting.We examine significance with a repeated mea-

sures ANOVA on each experimental cell, with onebetween-subject factor, panel (four levels); and twowithin-subject factors, estimation method (four levels)and question-selection method (two levels).16 Deter-ministic polyhedral question selection predicts sig-nificantly better than orthogonal question selection:across panels (p < 007) and within the Australian

16 Similar significance levels were obtained with an ANOVA similarto the ANOVA for the Table 2 data, with an additional between-subject factor capturing the experimental cell.

panel (p < 005 – ANOVA on the Australian panelonly). Probabilistic polyhedral question selection per-forms significantly better than aggregate customiza-tion question selection: overall (p < 006), across thenon-U.S. panels (p < 002), and in the New Zealandpanel (p < 005). Probabilistic polyhedral questionselection is better in three of the four panels and neversignificantly worse. While the results do not alwaysobtain a significance level of 0.05, we can say, at min-imum, that probabilistic polyhedral methods showpromise.In summary, the proposed probabilistic polyhe-

dral question-selection methods improve correlationsbetween predicted and actual choice in at least somesituations. HB remains the best estimation methodoverall. Both ACe and ACi improve predictive abilityrelative to deterministic polyhedral methods (AC).

Substantive Results: Consumer Reactions toStelvin Screw Caps for Fine WineFigure 5 reports the estimates of the average part-worths for wine closures for leading-edge wine con-sumers in the United States, Australia, and NewZealand.17 In Australia and New Zealand there is aslight preference for Stelvins over traditional corkclosures. However, for the United States, corks arestrongly preferred to Stelvins and closure type is amore important attribute. U.S. consumers even preferMetaCorks™ and synthetic corks to Stelvins, whereasAustralians and New Zealanders prefer Stelvins tothese other nontraditional closures.We also examine the importance of wine closures

relative to other features. Figure 6 reports the aver-age partworths for the type of wine and the originof the wine. Preferences for the type of wine and thecountry of origin are roughly the same for U.S., NewZealand, and Australian consumers, with the excep-tion of a home-country bias. (Detailed partworth val-ues are available from the authors.)The data suggest that for U.S. consumers, the rel-

ative importance of Stelvins versus corks (6.91) is

17 The average partworths have been normalized such that the low-est level of each attribute has a partworth of 0 and the sum of thepartworths across attributes is 100.

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


Table 3 Comparing Question Selection Methods—Correlation with Choice

Australian New Zealand Average of First U.S. Second Average ofExperimental cell Question-selection method panel panel Australia and NZ panel U.S. panel U.S. panels

Deterministic polyhedral vs. Deterministic polyhedral 0492 0429 0468 n/a n/a n/aorthogonal (n= 527) Orthogonal 0445 0405 0430 n/a n/a n/a

Probabilistic polyhedral vs. Probabilistic polyhedral 0498 0457 0483 0373 0282 0300aggregate customization Aggregate customization 0470 0411 0449 0385 0270 0293(n= 1728)

Figure 5 Average Partworths for Wine Closures in the U.S., Australia,and New Zealand Studies

0

1

2

3

4

5

6

7

8

Cork Synthetic cork MetaCork Stelvin

Ave

rage

par

twor

ths US AUS NZ

comparable or less than the relative importance ofwine type (dry red versus blush red, 17.88), region(United States versus France, 6.68), and type of win-ery (regional versus international, 8.36). At least ini-tially, bottles with Stelvin closures will have to beoffered at a discount in order to capture a significantmarket share. For example, for higher-priced wines,market simulations based on our estimates (aver-aged across question-selection methods) suggest that

Figure 6 Average Partworths for the Type of Wine and Wine Origin

Type of wine

02468

101214161820

Dry red Dry white Aromatic white Blush red

Ave

rage

par

twor

ths

US AUS NZ

Wine origin

0

5

10

15

20

25

Australia/NZ France Sonoma/Napa Chile/Argentina

Ave

rage

par

twor

ths

combining Stelvins with a $10 discount would allowcapturing a 42.1% market share.18 Based on theseresults, it appears that (1) there is current resistance toStelvins among leading-edge U.S. consumers, (2) theimportance of closure is not high relative to other fea-tures of wine, and (3) for current U.S. leading-edgeconsumers, a modest price discount might encouragethe adoption of Stelvins.

Do the Managerial Recommendations ChangeBased on Question-Selection Method?As an illustration, Figure 7 plots the estimates of theaverage partworths for wine closures based on thetwo question-selection methods in the U.S. panels.Comparing the two plots, we see subtle differencesbetween methods. However, these plots only capturethe average partworths across respondents, not thefull distribution of partworths estimates. Moreover,without reference to the managerial context, it is diffi-cult to intuit whether the estimates based on differentquestion-selection methods imply differences in strat-egy. Thus, we examine the quantitative implications.We begin by comparing the predicted response to

a price discount of $10 on Stelvin closures. Partworthestimates based on aggregate customization questionssuggest that a price discount of $10 on Stelvin clo-sures would capture 44.3% of this premium wine mar-ket; partworth estimates based on the probabilisticpolyhedral questions suggest lower market share of39.8% (p < 003). Depending on the costs of market-ing Stelvins, this lower reward might be the differencebetween a GO and a NO GO decision.To gain further insight into whether or not part-

worth differences imply different managerial deci-sions, we draw on recent research by Belloni et al.(2005). Belloni et al. solve an optimal product designproblem based on partworth data similar in structureto that collected here. Their design problem consistsof selecting product features for the profiles in a prod-uct line in order to maximize profit (faced with a fixedset of competitors). Using Lagrangian relaxation withbranch and bound, they identify the optimal product

18 Results are approximate because the sponsors defined prices withranges. We used the midpoint of each range in our share and profitcalculations.

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


Figure 7 Average Partworths for the Wine Closures by Two DifferentMethods

Aggregate customization

01234567


Ave

rage

par

twor

ths

Ave

rage

par

twor

ths

US AUS NZ

Probabilistic polyhedral

01234567


line for relatively large numbers of features and cus-tomers. More importantly, they compare a varietyof heuristics and demonstrate that simulated anneal-ing (1) is feasible for reasonably sized problems and(2) achieves 100% of the optimum in their test prob-lems (p. 20). We adopt their structure and modifytheir simulated annealing code to optimize a productline of wine profiles based on cost estimates obtainedfrom wine experts. We assumed that the competi-tive products available to consumers were the set ofall profitable profiles containing traditional corks. Weassumed that each consumer purchased exactly onebottle.Using Belloni et al.’s (2005) formulation, we devel-

oped optimal Stelvin-based product lines (consistingof 10 products) using (1) probabilistic polyhedralquestions and (2) aggregate customization questions.Within this framework, the product line based on thepartworths obtained by probabilistic polyhedral ques-tions had three profiles in common with that designedbased on the aggregate customization partworths.Another four pairs of profiles varied on one feature.Furthermore, if the partworths obtained from proba-bilistic polyhedral questions describe the market, theprofit obtained with the polyhedral-based productline was 19.4% higher than the product line based onaggregate-customization partworths.19

19 This calculation assumes that probabilistic polyhedral is the bestestimate and is provided for illustration. Profit is guaranteed to beno worse by the principle of optimality. However, we find the mag-nitude of the difference—almost 20%—to be interesting, especiallycompared to the 2.1% difference due to optimization method foundby Belloni et al. (2005, p. 28). For alternative methods of profitabil-ity comparisons, see Rust and Verhoef (2005).

7. Conclusions and Future ResearchThis paper focuses on improved methods for adap-tive question selection in conjoint analysis. We nestdeterministic polyhedral methods using conjugateclasses of likelihood functions and prior distribu-tions. Our probabilistic Bayesian framework over-comes prior weaknesses by enabling researchers to(1) take response error into account and (2) intro-duce informative priors. Simulation and empiricaltests suggest that these improvements are promis-ing. The wine-closure application is the first predic-tive test of choice-based polyhedral methods. For thisapplication, individual adaptation of questions showspromise.We close by noting limitations and avenues for

future research. First, computation issues forced usto use approximations and to use prior distribu-tions described by only few polyhedra. More effi-cient algorithms could be developed and the structureof the problem may be exploited further to allevi-ate this limitation. Second, analytic-center estimationcontinues to improve, but does not yet perform aswell as hierarchical Bayes. Using the Bayesian inter-pretation of polyhedral question selection, we mightderive a formal Bayesian-loss-function minimizationthat improves analytic-center estimation (Rossi andAllenby 2003). Third, as a first approximation weused the same value of for all choice questionsand all respondents. We might improve estimation if is specified as a function of the difficulty of thechoice questions and/or is allowed to vary overthe number of questions (Liechty et al. 2005). Usingthe formulae in this paper, we might also specifya prior on , conditional on the partworths andthe choice set, and formulate a posterior given theobservations.20 Fourth, our simulations used a now-standard structure, but there remain interesting testswith nondiagonal covariance matrices and specifi-cations where the average partworths vary. Finally,other approaches to handling response error may bedeveloped using stochastic optimization (Spall 2003)or statistical learning theory (Evgeniou et al. 2005).Polyhedral methods remain a nascent technique thatwe hope will improve with future testing and futuredevelopments.

AcknowledgmentsThis research was supported by the MIT Sloan Schoolof Management, MIT’s Center for Innovation in ProductDevelopment, and Northeastern University’s Institute forGlobal Innovation Management. The authors wish to thankKevin Stanik and Andrew Rutkiewitz for their computersupport, as well as Tom Atkin and Larry Lockshin with

20 We would like to thank the AE for suggestions regarding theparameter .

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


their wine industry expertise. Michael Yee has providedvaluable comments that the authors gratefully acknowl-edge. They also thank Matthew Selove for suggesting theproduct-line optimization test.

Appendices. Derivations and Algorithms

A1. Computation of ′, the Tuning Parameter forError Modeling

We compute ′ as follows:• Estimate a population mean for the partworths, upop.

In Monte Carlo simulations we use the true mean; in theempirical applications we use hierarchical Bayes estimatesfrom the pretest subjects.• Generate R random questions (R= 100 in the simula-

tions and experiments) with logistic probabilities based onupop. The probability, ′

r , that a respondent chooses the max-imum utility profile is the maximum logistic probability forthat respondent on that question. Averaging over respon-dents gives an initial estimate, ′

o.• Use upop to simulate N respondents (N = 100 in the

simulations and experiments) using o = J ′o − 1/J − 1

for polyhedral question selection. Recompute ′ as above(assuming logistic probabilities).In theory, one might iterate these steps toward conver-

gence; however, in practice we found that ′ was not sen-sitive to the initial estimate, o used to generate the ques-tions. Nonetheless, this means that our simulations areconservative.

A2. Sensitivity to ′In the simulations, we purposefully base the choices of syn-thetic respondents on logistic probabilities. These simulatedchoices imply that ′ varies by respondent and thus tests thesensitivity of our approximation that ′ is constant acrossrespondents. We study further the sensitivity with respectto ’ using a simulation set up similar to that in §4. We sim-ulate five sets of 100 synthetic respondents using magnitudeand heterogeneity parameters equal to 2.0. Questions areselected with probabilistic polyhedral methods with errormodeling and informative priors; the estimation method isACe+i.Figure A.1 suggests that predictive accuracy is flat for a

fairly wide interval for ′. Fortunately, the values obtainedusing the above procedure always fell within this interval

Figure A.1 Sensitivity of Predictive Accuracy to ′

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

′

RM

SE

for both the simulations and the empirical tests. Predictionsdegrade only as ′ is selected to imply almost randomchoice (toward 0.3) or almost no response error (toward 1.0).

A3. Mixture Weights for Informative Priors(Mixtures of Normal Distributions)

Our goal is to approximate a normal distribution N u by amixture of uniform distributions supported by polyhedra:

M∑m=1

mPm u+

(1−

M∑m=1

m

)P0

u

where m = u u − Cm ≤ u ≤ u + Cm, u ≥ 0, e′ u = 100. Inthe Monte Carlo simulations, we used M = 3, with C1 = 5,C2 = 10, and C3 = 15. Without loss of generality, assume thatC1 <C2 < · · ·<CM so that the “boxes” used to approximatethe normal distribution are of increasing sizes. The weights1 M are found by solving the following system ofequations:

Prob u ∈1 u∼N u=1+2Prob u∈1 u∼P2

+···+MProb u∈1 u∼PM

+(1−

M∑m=1

m

)P u ∈1 u∼ P0

Prob u ∈2−1 u∼N u=2Prob u∈2−1 u∼P2+3Prob u∈2−1 u∼P3

+· · ·+MProb u ∈2−1 u∼ PM

+(1−

M∑m=1

m

)P u ∈2−1 u∼ P0

Prob u ∈M −M−1 u∼N u=MProb u ∈M −M−1 u∼ PM

+(1−

M∑m=1

m

)P u ∈M −M−1 u∼ P0

In the Monte Carlo simulations the left-hand sides wereapproximated numerically by drawing 10,000 sets of para-meters from N u, where N u was the distribution used togenerate the true partworths. Prob(u ∈m −m−1 u∼ P0

was computed numerically by drawing 10,000 sets ofparameters from P0

. We recognize that Prob(u ∈ m −m−1 u∼ Pm

is equal to

Prob u ∈m −m−1 u∼ P0

Prob u ∈m u∼ P0

The numerator and denominator were computed numeri-cally.

A4. Incorporating Population Priors forFeature Importances

In our empirical application, all features have the samenumber of levels, so we define importance as the sum ofthe partworths for that feature (setting the lowest partworthto zero). Importance can also be defined as the differencebetween the highest and lowest partworth for a feature.

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.


We have found that constructing priors based on constraintson the importances is more practical and intuitively appeal-ing than using constraints on the partworths themselves.Let mf be the median of the importance of feature f

based on individual-respondent estimates. Let mf be themedian importance based on P0

u and let Pu be the proba-bility that the importance of feature f is smaller than mf forP0

u. Compute Pu numerically with 10,000 draws on 0.If mf > mf ( Pu > 05, the constraint corresponding to fea-ture f is that its importance is greater than mf . This con-straint is associated with the weight 3f , such that 05= 3f +1− 3f 1− Pu. If mf ≤ mf Pu ≤ 05, the constraint cor-responding to feature f is that its importance is less thanmf , and the corresponding weight is 3f such that 05= 3f +1− 3f Pu.Let F be the number of features, and SF be the set of

subsets of all subsets of 12 F . The prior distributionis P u=∑

s∈SF msPs

u, where• ms

=∏Ff=1 3

f∈sf 1− 3f

fs where f ∈ s is 1 if f is in sand 0 otherwise; f s is its complement.• s is the polyhedron obtained from adding the con-

straints corresponding to the features f ∈ s to the initialconstraints defining 0.

A5. Summary of Probabilistic Polyhedral QuestionSelection

1. Compute the weights for the probability mixture, ws =s1−i−s, for all s ∈ Sq .

2. Compute the analytic center of the mixture, uq =∑Mm=1

∑s∈Sq mwsACs ∩m.

3. Approximate each polyhedron s ∩m with an ellip-soid and compute the longest axis of the ellipsoid accordingto deterministic polyhedral methods (see THS for details).4. Solve for the eigenvalues of V ′)V and select the J/2

eigenvectors associated with the largest J/2 eigenvalues.(If J is odd, find the J + 1/2 longest axes.) See the section“Longest Axes of a Mixture of Polyhedra” for details.5. Find the intersections of the longest axes of the prob-

ability mixture with 0 ⇒ uj .6. Find the J profiles by solving the knapsack problem,

maximize xj uj subject to xjuq ≤ K, where K is a randomly

drawn constant. (See THS for details.)21

21 In this paper, we drew K up to 30 times until all profiles weredistinct. If all profiles are identical after 30 draws, it is likely that nofurther questions are needed and the questioning sequence stops.If after 30 draws there are only K ′ distinct solutions (1 < K ′ < K),these are presented to the respondent.

ReferencesAilawadi, K. L., P. K. Kopalle, S. A. Neslin. 2005. Predicting com-

petitive response to a major policy change: Combining game-theoretic and empirical analyses. Marketing Sci. 24(1) 12–24.

Allenby, G. M., N. Arora, J. L. Ginter. 1995. Incorporating priorknowledge into the analysis of conjoint studies. J. MarketingRes. 32(May) 152–162.

Arora, N., J. Huber. 2001. Improving parameter estimates andmodel prediction by aggregate customization in choice exper-iments. J. Consumer Res. 28(September) 273–283.

Belloni, A., R. Freund, M. Selove, D. Simester. 2005. Optimiz-ing product line designs: Efficient methods and comparisons.Management Sci. Forthcoming.

Courtney, S. 2001. Screwcap wine seals: Has Kiwi ingenuity gonetoo far?Wine of the Week (August). http://www.wineoftheweek.com/screwcaps/history.html.

Evgeniou, T., C. Boussious, G. Zacharia. 2005. Generalized robustconjoint estimation. Marketing Sci. 24(3) 415–429.

Hauser, J. R., O. Toubia. 2005. The impact of utility balance andendogeneity in conjoint analysis. Marketing Sci. 24(3) 498–507.

Huber, J., K. Zwerina. 1996. The importance of utility balance inefficient choice designs. J. Marketing Res. 33(August) 307–317.

Kanninen, B. J. 2002. Optimal design for multinomial choice exper-iments. J. Marketing Res. 36(May) 214–227.

Liechty, J. C., D. K. H. Fong, W. DeSarbo. 2005. Dynamic mod-els incorporating individual heterogeneity: Utility evolution inconjoint analysis. Marketing Sci. 24(2) 285–293.

Liu, Q., T. Otter, G. M. Allenby. 2006. Investigating endogeneitybias in marketing. Marketing Sci. 26(5) 640–648.

Lockshin, L., P. Quester, T. Spawton. 2001. Segmentation byinvolvement or nationality for global retailing: A cross-nationalcomparative study of wine shopping behavior. J. Wine Res.12(December) 223–236.

Rossi, P. E., G. M. Allenby. 2003. Bayesian statistics and marketing.Marketing Sci. 22(3) 304–328.

Rust, R. T., P. C. Verhoef. 2005. Optimizing the marketing inter-ventions mix in intermediate-term CRM. Marketing Sci. 24(3)477–489.

Spall, J. C. 2003. Introduction to Stochastic Search and OptimizationEstimation Simulation and Control. Wiley-Interscience, Hobo-ken, NJ.

Toubia, O., J. R. Hauser, D. I. Simester. 2004. Polyhedral methodsfor adaptive choice-based conjoint analysis. J. Marketing Res.41(February) 116–131.

Toubia, O., D. Simester, J. R. Hauser, E. Dahan. 2003. Fast polyhe-dral adaptive conjoint estimation. Marketing Sci. 22(3) 273–303.

INFORMS

holds

copyrightto

this

article

and

distrib

uted

this

copy

asa

courtesy

tothe

author(s).

Add

ition

alinform

ation,

includ

ingrig

htsan

dpe

rmission

policies,

isav

ailableat

http://journa

ls.in

form

s.org/.

ProbabilisticPolyhedralMethodsfor AdaptiveChoice …hauser/Papers/Toubia_Hauser_Garcia... ·...

Documents

Transcript of ProbabilisticPolyhedralMethodsfor AdaptiveChoice …hauser/Papers/Toubia_Hauser_Garcia... ·...