Allowing for Uncertainty in Choice Experiments

Dolores Garcia ∗ Departament d'Economia Aplicada

Universitat de les Illes Balears E-07122 Palma de Mallorca, Spain

e-mail: dolores.garcia@uib.es

David Hoyos Instituto de Economía Pública

Edificio Central, 4ª. Planta Avda. Lehendakari Aguirre, 83

48015 Bilbao e-mail: david.hoyos@ehu.es

Pere Riera Departament d’Economia Aplicada Universitat Autònoma de Barcelona

Edifici B, Campus UAB 08193 Bellaterra, Spain

e-mail: pere.riera@uab.es

January, 2008

Abstract

In recent years choice modeling methods have gained popularity among the group of valuation techniques used to elicit the population’s preferences on non-market goods. Despite the large number of available applications, to our knowledge available studies have not explicitly addressed the problem of WTP uncertainty. This paper is based on the results of an exercise that combines a choice experiment and a contingent grouping format. Follow-up questions were distributed in two sub-samples in order to capture either the degree of certainty or the level of difficulty of the choice set, as perceived by respondents. Also, the time spent to fulfill the tasks was accounted for, for each of the choice sets faced by respondents. The main results are reported, comparing choice and contingent grouping elicitation formats. Also, we investigate whether marginal values are affected by reported uncertainty. The exercise was applied in the Basque Country and Navarra in order to value a set of key environmental attributes representative of the natural area of Jaizkibel. JEL Classification: Q51

∗ Corresponding author

1. Introduction

The valuation of environmental goods has received a great deal of attention in

the last decades. Among the methods used for this purpose, stated preference (SP)

techniques have become very popular, and numerous methodological advances have

taken place.

One of the topics that has received attention is how uncertainty might affect

estimated values. Thus, uncertainty might be present in the definition of the provided

good, the effects of a particular environmental policy, or even in the person’s

knowledge about her true willingness to pay for the environmental change (Li and

Mattsson, 1995). Several studies have dealt with this topic in the CV literature (Berrens

et al., 2002; Burton et al., 2003; Alberini et al., 2003; Vossler and Poe, 2005). The usual

way of incorporating uncertainty into respondents’ answers is to allow them to state the

degree of certainty with which they answered. Depending on their answers, then

responses are included or not in the estimation process that leads to the calculation of

marginal values.

This approach has been considered in this paper. A choice modeling framework

has been developed to value a set of environmental attributes characteristics of mount

Jaizkibel, a natural area on the Cantabrian Sea, near the French border (Bateman et al,

2002). In particular, both a choice experiment and a contingent grouping exercise are

developed. Choice experiments have been widely applied for the valuation of

environmental amenities, and are probably the most popular among the group of choice

modeling methods. A contingent grouping approach was also used (Brey et al, 2005).

Under the latter, individuals are asked to group the alternatives in sub-sets of those

which are better that the status quo or business-as-usual alternatives, and those which

are worse.

Follow-up questions were included to obtain information about the degree of

certainty as perceived by respondents, or alternatively, about the difficulty when

responding to the choice set scenario. We analyze the results of such questions, and

discuss the extent to which every single respondent discriminates in their answers, how

reported answers vary with the particular elicitation format used, and how marginal

values are modified when uncertainty is accounted for.

The structure of the paper is as follows. Next section briefly describes the site

object of the valuation exercise, and the survey design. In section 3 the main results of

the debriefing questions are presented, with three subsections: that devoted to the

certainty question, another one dedicated to the difficulty question, and a final

subsection in which how times of responses vary under different scenarios are given.

Section 4 first provides the results of a main effects conditional logit model, then

discuss the new estimation outcomes when the result of the certainty question is

interacted with the cost attribute and included as explanatory variable. The final section

includes the main conclusions of the paper.

2. Description of the exercise 2.1. Description of the site and attributes

The application was developed in order to value the relevant environmental

characteristics of mount Jaizkibel, a 2.400 hectares natural site that contains 15 zones

declared of high ecological interest by the European Union, situated in the municipality

of Pasaia (Basque Country). The environmental characteristics of this site might be

affected if the project to build a new port in the outside of the bay of Pasaia, under the

hills of mount Jaizkibel, is undertaken. According to a recent study, the construction of

the new seaport would provoke some critical impacts (Pozueta 2004), including cliffs

destruction, loss of vegetable cover and loss of seabed and local beaches, among other.

Detailed information about the environmental characteristics of Jaizkibel can be found

in Hoyos et al. (2008)

In order to identify the key environmental attributes of mount Jaizkibel and the

business-as-usual levels of provision, interviews with experts and focus groups were

undertaken. It was concluded that the most important ones were landscape, flora,

avifauna and seabed. In particular, landscape was defined as the percentage surface

from which today’s landscape could be seen in the future; flora was measured by the

future level of protection of today’s population of armeria euskadiensis, an endemism

of basque seacoast; avifauna, described in terms of the future level of protection of

today’s population of lesser and peregrine falcon; and seabed, measured by the future

level of protection of today’s extension of red algae. Pictures were used to illustrate the

different levels of all four attributes (see figure 1). To decide upon the levels of the cost

attribute, focus groups together with a pilot survey including open-ended contingent

valuation questions were used. The payment consisted of an annual contribution to a

Foundation exclusively dedicated to protecting mount Jaizkibel, that all Basque citizens

would be required to make. Table 1 summarizes the levels of all the attributes used.

Table 1. Attributes and levels considered

Attribute Level

Landscape 40%* 60% 80% 100%

Flora 50%* 70% 85% 100%

Fauna 25%* 50% 75% 100%

Seabed 50%* 70% 85% 100%

Annual payment 5 € 10 € 15 € 20 € 30 € 50 € 100 €

*BAU alternative levels

Combining all these attributes and levels, near two thousand different

combinations were obtained (44x71). As it is usually done when the universe of

alternatives is very large, statistical design methods were used to simplify the choice

sets construction (Louviere, Henser, and Swait, 2000). A main effects fractional

factorial design with second order interactions reduced the number of alternatives to 96

pairs of protection alternatives. The profiles were grouped in 24 blocks of four sets

containing two alternative protection programmes (programs A and B), plus the

business as usual option. Also, the “don’t know” option was included in order to avoid

the “yea saying” bias (Arrow et al. 1993). Then, each individual was shown 4 screens.

Two of them asked for the most preferred alternative (choice), and the remaining two

required their grouping as explained in the introductory section. The order in which

choice and grouping were combined was randomly picked.

2.2. Questionnaire and data collection

A questionnaire was developed to simulate a market in which individuals would

be willing to choose among varying levels of protection of the attributes. The

questionnaire was finally structured in three parts. The first part described the main

attributes and, the current situation of mount Jaizkibel and the potential future damages

to its environmental attributes. The second part (preference elicitation part) contained

the choice experiment and contingent grouping questions. An example of a card

including a choice set and the screen the interviewee saw is shown in figure A.1 in the

annex.

The last section collected some debriefing and socioeconomic questions,

including those aimed at investigating about the degree of certainty in responses and the

perceived level of difficulty when choosing or grouping alternatives. Thus, after each

set of alternatives had been shown, a follow-up question was added. A split sample

approach was used. Half the sample was asked to state, in a scale from 1 to 7, how sure

they felt when making their choice or groups. The other half was asked to state how

difficult the choice or grouping had been, again in a scale from 1 to 7.

Interviews were conducted face-to-face at people’s homes, using laptop

computers. Respondents could read the screens and listen to a recorded voice. Answers

could be typed in by themselves or dictated to the interviewer, at respondents’

discretion. This way of administering the questionnaire allowed for the possibility of

collecting information about the time individuals took to make their choices or grouping

decisions. The relevant population considered was the population from the Basque

Autonomous Community and Navarra in Spain as well as some French cities next to the

Spanish border, accounting for 2.5 million people being at least 18 years old. The pilot

was conducted in October 2006, while the final survey was undertaken between

November and December, 2006. A stratified random sample of 636 individuals was

selected from the relevant population. The strata used included age, gender and size of

the town of residence, following official statistical information (EUSTAT). In each

location, the questionnaires were distributed using random survey routes.

3. Descriptive statistics

The sample accurately represented the population in terms of age, gender and

income levels. Even though the questionnaire included a number of questions asking for

the respondents’ view on the importance of the included attributes, their involvement in

recreational activities, and many other, here we mostly focus on the results of the

follow-up questions dealing with uncertainty and difficulty issues. For other outcomes,

see Annex II of Hoyos et al. (2007).

3.1. Certainty in choices and grouping tasks

The follow-up question asked the respondent how “sure” she felt with the

choice/grouping just provided, in a scale of 1 to 7, 7 representing the highest level of

security. The means for the four follow-up questions resulted in values comprised

between 5.5 and 6, representing a medium-high level of confidence. The overall mean

was 5.75 (Table 2), and responses did not vary between the choice and the grouping

elicitation formats.

Dummy variables were created in order to capture whether the respondent had

felt sure about the response (when answered 5, 6 or 7) or unsure (when answered 1,2, 3

or 4).

Table 2. Degree of certainty . Screens 1 to 4, and average.

Mean (Std.Dev.)

(values 1 to 7)

% of unsure responses

1st screen 5.78 (1.57) 16.98%

2nd screen 5.63 (1.55) 22.28%

3rd screen 5.79 (1.44) 15.66%

4th screen 5.84 (1.44) 14.45%

Average 5.76 (1.50) 18.30%

When analyzing individual responses, it was found that a 54.61% of respondents

did not change the value of certainty with the cards, that is, they picked the same value

for all four choice sets. The remaining 45.39% did provide a different value depending

on the particular card just seen. Mean values when considering those who do and who

do not discriminate in their certainty answers are different. In average, those individuals

who do modify the degree of certainty in response show a smaller mean and a larger

standard deviation (table 3).

Table 3. Degree of certainty according to individuals’ responses

Respondent answered…

Mean (Std.Dev.)

(values 1 to 7)

% of unsure responses

Different scores 5.14 (1.62) 31’88%

Same scores 6.26 (1.22) 6,02%

When working on the experimental design, dominated profiles and profiles’ sets

in which attributes’ levels did not vary tried to be eliminated or reduced (Huber and

Zwerina, 1996). However, a small number of choice sets remained in which the levels

of some attribute were the same. We describe these sets as easy. Although they

represent a small number, it results that the mean degree of certainty in responses is

higher (6), compared to difficult sets (5.75).

Table 4. Degree of certainty depending on the difficulty of the choice set

Profiles’ set

Mean (Std.Dev.)

(values 1 to 7)

% of unsure responses

Easy 6 (1.14) 17’64%

Difficult 5.75 (1.51) 18,32%

3.2. Difficulty of choices and grouping tasks

As shown in the annex, some respondents faced follow-up questions after the

choice and grouping screens, asking for the degree of difficulty of the tasks of choosing

or grouping the alternatives, and the answer again followed a closed format, in a scale

going from 1 to 7 (1 meaning the task had been very easy, 7 meaning it had been very

difficult). The main results are reported here.

Table 5 summarizes the mean scores from the first to the fourth task. There was

little variation in average means, results did not change either when grouping and choice

elicitation formats were compared. The average score resulted in 3.33, and so tasks

could be considered as moderately easy.

Answers were also recoded in a binary way: when the stated degree of difficulty

was 1,2, 3 or 4, the task was considered not difficult, and difficult for scores above 4.

The percentage of difficult was above 37% for the first choice set, and diminished as the

subsequent screens were shown, resulting in 31.68% for the last choice set.

Table 5. Degree of difficulty. Screens 1 to 4, and average.

Mean (Std.Dev.)

(values 1 to 7)

% of difficult responses

1st screen 3.41 (2.13) 37.12%

2nd screen 3.32 (2.08) 32.67%

3rd screen 3.39 (2.13) 32.17%

4th screen 3.27 (2.13) 31.68%

Average 3.33 (2.12) 33.17%

With respect to the certainty question, the percentage of individuals who did not

keep the same answer for all four choice sets was larger, achieving a 62.69%. Only the

remaining 37.31% answered the same in all instances. The reported average degree of

difficulty was

Table 6. Degree of difficulty according to individuals’ responses

Mean (Std.Dev.)

Respondent gave… (values 1 to 7) % of difficult responses

Different scores 3.39 (1.93) 34.50%

Same scores 3.19 (2.42) 32.30%

Finally, it would be expected that easy choice sets, as described in the previous

subsection, would result easier for respondents. As table 7 shows, this was the case for

choice tasks; however, for contingent grouping tasks the average reported score resulted

higher for presumably easier tasks.

Table 7. Degree of difficulty depending on the difficulty of the choice set

Profiles’ set

Mean

(values 1 to 7)

Choice experiment Contingent grouping

Easy 3 3.53

Difficult 3.4 3.27

3.3. Times of response

As explained in the previous section, we used a computer-aided questionnaire in which

responses were automatically stored in a dataset, and in which the time spent in

choosing from or grouping the alternatives was stored.

The average amount of time used to fulfill the task of choosing or grouping the

alternatives was slightly above 27 seconds. Grouping the alternatives took an average of

29’8 seconds, while for choices 24’8 seconds were required. The histogram in figure 1

represents the percentage of responses for different time intervals.

Figure 1. Histogram for time intervals

05

1015202530354045

<15seconds

15-30seconds

30-45seconds

45-60seconds

60-120seconds

> 2minuts

% of cases

In almost 40 per cent of the cases respondents answered in less than 15 seconds,

and this proportion rises to more than 70 per cent for those under the 30 second

threshold. More than 90 per cent of cases individuals invested less than 1 minute in

fulfilling the task of choosing or grouping.

Also, the time of response clearly diminished as choice sets were presented. For

the first choice of profiles shown, individuals took an average of 45 seconds to answer,

while they invested about one third of such time for the last choice set (17 seconds). The

corresponding standard deviations diminish as well. Table 8 summarizes some of the

commented results.

Table 8. Time of response for choice/grouping screens 1 to 4, and average

Mean (Std.Dev.)

(figures in seconds)

1st screen 45.03 (51)

2nd screen 25.91 (32)

3rd screen 20.38 (21)

4th screen 17.09 (16)

Grouping

29.75 (39)

Average

27.40 (34.27) Choices

24.84 (31)

Combining the fact that respondents were progressively faster when providing

their responses, and that they invested some more time in the grouping of alternatives,

the times of response have been compared. Thus, average times of response have been

found for grouping and choosing tasks, differentiating the cases in which individuals

first faced the grouping screens and then the choice ones, or the other way around. The

following table summarizes the results.

Table 9. Time of response (seconds) depending on task and sequence of of tasks

Mean (Standard deviation)

Choice 1st, Grouping 2nd Grouping 1st, Choice 2nd

Choice 32.26 (38) 16.79 (15)

Grouping 20.58 (22) 39.22 (49)

As expected, in all instances the first two tasks take more time than the second

ones. The reduction of time is greater when the grouping screens appear first, with

average times going from almost 40 seconds (for grouping) to around 17 (for choice).

The shortest times correspond to the case in which choices appear in the second place

(approximately 17 seconds), and the longest to the instance in which grouping screens

go first (39.22 seconds).

Finally, times of response were compared taking into account whether the

profiles’ set was easy or not. There was a small difference of 1.31 seconds in favor of

difficult sets, which took less time. This somehow unexpected result is driven by

differences in times for choice tasks: grouping tasks were performed quicker when the

set of alternatives were “easy”, but with choice tasks it happened the other way around.

4. Estimation results considering uncertainty

In order to carry out the estimation of the values of the attributes at stake, the

database was depurated. In this section, only choice responses are considered. An 8 per

cent of interviewees declared not having understood the objective of the program to

preserve mount Jaizkibel, and were not shown the choice sets. Similarly, protests to the

valuation questions took place for 171 individuals, representing the 26.8% of the total

sampled population. After all the aforementioned rejections, 415 observations out of the

original 637 remained valid for estimation purposes. Finally, since a split sample

procedure was used to include follow-up questions, the number of valid observations in

which certainty questions have been included diminishes to 304.

As explained above, the attributes included in the choice sets were the

percentage values of landscape preservation, flora, avifauna and seabed. All of them

were treated as continuous variables. A conditional logit model was specified, including

the attributes in the terms explained above, and considering all main effects. The model

including a constant for the business-as-usual alternative was also considered, but the

restricted model was accepted according to the test of the likelihood ratio test.

Table 10. Conditional logit model (model 1)

Attribute Coefficient (t-ratios)

Landscape 0.0246 (5.786)

Flora 0.0160 (3.074)

Avifauna 0.131 (3.622)

Seabed 0.0070 (1.813)

Payment -0.0206 (-5.705)

Log-likelihood function: -243.6517

Observations: 304

The results show that all variables resulted significant (for seabed, the

coefficient is significant at a 10% significance level), and the expected signs were

found.

A second model specification including information on the reported certainty of

response was included, for the exact same observations. The reported scored was

recoded into a binary variable capturing whether the choice had been made with

certainty or not, as explained above. Then, this variable was interacted with the cost

payment. The objective was to see whether uncertainty could help explain choices, in

the first place, and see how the resulting marginal values would be affected. The results

of the estimation appear in table 11.

Table 11. Conditional logit model (model 2) considering uncertainty

Attribute Coefficient (t-ratios)

Landscape 0.0246 (5.796)

Flora 0.0161 (3.087)

Avifauna 0.0137 (3.774)

Seabed 0.0072 (1.873)

Payment -0.0185 (-5.034)

(Payment)*(choice is uncertain) -0.0191 (-1.858)

Log-likelihood function: -241.6445

Observations: 304

Again all attributes’ coefficients result significant, and so results the interaction

including certainty of the choice made. The values corresponding to WTP for each

attribute, or implicit prices, could be calculated out of the coefficients in table 11. They

are included in the following table.

Table 12. Annual WTP for Jaizkibel attributes (in € of 2007/person-year)

Model 1 Model 2 Marginal WTP Attribute

Marginal WTP Certain Uncertain

Landscape 1.194 1.330 0.654 Flora 0.777 0.870 0.428 Avifauna 0.636 0.741 0.364 Seabed 0.340 0.389 0.191

The values included in the table correspond to 1% marginal changes in the

attribute’s preservation. The values corresponding to model 1 would be the resulting

ones when all choices are considered, and the information on the declared level of

certainty in the response is ignored. When using the information on whether the choice

was a sure one or not, willingness to pay is affected by certainty. Thus, when choices

are certain, willingness to pay is larger for all four attributes. In other words, deleting

the “uncertain” responses from the dataset from which the models are estimated would

lead to greater marginal WTPs.

5. Conclusions

This paper constitutes an attempt to include information on the degree of

certainty of individuals, when making their choices, into the statistical analysis of data

resulting from a choice modeling exercise. The incipient results suggest that values

might result affected by feeling of confidence in the responses, and this poses the

question of whether choices should be included when respondent do not feel sure about

them. Variation in resulting values might be of importance when calculating welfare

changes, especially at a more aggregate level.

Other interesting issues have to do with the interpretation of uncertainty and how

it might relate to other aspects of the exercise, including the level of difficulty of the

task at stake. The results show that respondents felt rather sure about their responses,

and that they found the tasks easy enough not to be considered difficult (in average). On

the other hand, the percentages of instances in which the tasks were fulfilled with

uncertainty were considerably lower than the proportion of tasks considered difficult.

While both types of responses could be related, the issue needs further investigation.

The results were obtained from different sub-samples, too, so comparisons are not

straightforward. Likewise, the relationship between the times of response and the degree

of uncertainty in the response should be analyzed.

The analysis here introduced had the drawback, however, that the final model

specifications were made on a relatively low number of observations, for the sake of

comparison, given that split samples were used for choice and contingent grouping

formats, and also for certainty and difficulty questions. An immediate object of further

research consists in extending the analysis by undertaking the joint estimation of choice

and grouping answers.

Also, more accurate tests on the significance of the differences in resulting

scores should be used to compare results dealing with uncertainty and difficulty. Also,

the exploitation of the information provided by recording the times spent to responding

might prove extremely useful to help understand the cognitive burden of alternative

choice modeling approaches such as contingent grouping and choice experiments.

Acknowledgements The authors acknowledge the financial support from the Department of Environment of the Basque Government, and from CICYT Project No. SEJ2004-00143/ECON and CEDEX project from the Spanish government.

Bibiliography

Alberini, A., K.Boyle and M.Welsh (2003) “Analysis of Contingent Valuation

Data with Multiple Bids and Response Options Allowing Respondents to Express

Uncertainty”, Journal of Environmental Economics and Management, vol. 45, pp. 40-

62

Arrow K. et al. (1993). “Report of NOAA Panel on Contingent Valuation”.

Bateman, I. J., Carson, R. T., Day, B., Hanemann, M., Hanley, N., Hett, T.,

Jones-Lee, M., Loomes, G., Mourato, S., Özdemiroglu, E., Pearce, D. W., Sugden, R.,

& Swanson, J. (2002). “Economic valuation with stated preference techniques: a

manual” Cheltenham, UK: Edward Elgar.

Berrens, R.P. et al. (2002) “Further Investigation of Voluntary Contribution in

Contingent Valuation: Fair Share, Time of Contribution, and Respondent Uncertainty”,

Journal of Environmental Economics and Management, vol. 44, pp. 144-68.

Brey R., O. Bergland O. and P. Riera (2005) “A contingent Grouping Approach

for Stated Preferences”, Working Paper #22/2005, Norwegian University of Life

Sciences.

Burton, A.C. et al. (2003) “An Experimental Investigation of Explanations for

Inconsistencies in Responses to Second Offers in Double Referenda”, Journal of

Environmental Economics and Management, vol. 46, pp. 472-89.

Hausman J. and D. McFadden (1984) “Specification tests for the multinomial

logit model”, Econometrica, vol 52, pp. 1219-1240.

Hoyos, D., P.Riera, M.C. Gallastegui, J. Fernández-Macho and D. Garcia (2008)

“Choice modelling applied to assessing environmental impacts of transport

infrastructures: the case of Pasaia’s new seaport, Spain”. Manuscript

Hoyos, D., P.Riera, M.C. Gallastegui, J. Fernández-Macho and D. Garcia (2007)

“Informe final: Valoración Económica del entorno natural del monte Jaizkibel”. Final

report for the Environmental Council of the Basque Government.

Huber, J., and K. Zwerina (1996). The importance in utility balance in efficient

choice designs. Journal of Marketing Research, 33, 307-317.

Louviere J., D. A. Henser, and J. Swait (2000) “Stated choice methods: analysis

and applications”. Cambridge University Press, Cambridge.

Li, C. and L. Mattsson (1995) “Discrete Choice under Preference Uncertainty:

An Improved Structural Model for Contingent Valuation”, Journal of Environmental

Economics and Management, vol. 28, pp. 256-69.

Pozueta J. (2004). “Estudio comparado de las alternativas de desarrollo del

Puerto de Pasajes en relación con su grado de impacto medio-ambiental y

sostenibilidad.” Departamento para el Desarrollo Sostenible. Diputación Foral de

Gipuzkoa.

Vossler, C.A. and G.L. Poe (2005) “Analysis of Contingent Valuation Data with

Multiple Bids and Response Options Allowing Respondents to Express Uncertainty: A

Comment”, Journal of Environmental Economics and Management, vol. 49, pp. 197-

200.

Annex

Figure A.1. Example of choice set in the valuation exercise

Figure A.2. Screen showing the questions about the degree of certainty in the response

Figure A.3. Screen showing the question about the degree of difficulty in the response