The Economic Benefits to Retailers from Customer ... · Journal of Retailing 92 (2, 2016) 147–161...

A

oooppOa©

K

dwaCaiatpaoVo

ct

(

h0

Journal of Retailing 92 (2, 2016) 147–161

The Economic Benefits to Retailers from Customer Participation inProprietary Web Panels

B.J. Allen a,∗, Utpal M. Dholakia b, Suman Basuroy a

a University of Texas at San Antonio, College of Business, 4.01.08 B, One UTSA Circle, San Antonio, TX 78249-0631, United Statesb Jesse H. Jones Graduate School of Business, Rice University, 6100 Main Street, Houston, TX 77005-1892, United States

Available online 6 January 2016

bstract

Proprietary retailer-sponsored web panels are growing in popularity. We examine specific ways in which retailers may benefit from the creationf web panels and participation of customers in them. A quasi-experiment conducted in the field with a large online retailer using eighteen monthsf data and a difference-in-differences estimation approach reveals significant economic benefits to the retailer from web panels beyond generationf useful customer information. Web panel participation increased number of purchases by panel customers by 17 percent, cross-buying by 14ercent, and customer profitability by 36 percent. Contrary to conventional wisdom advocating moratoriums for respondents, we find that for webanelists answering each survey (up to a maximum of 27 in nine months) led to incremental positive effects with no evidence of tapering or reversal.ur results extend prior research and managerial implications regarding the question behavior effect by showing that proprietary web panels yield

substantial positive economic impact for retailers. 2015 New York University. Published by Elsevier Inc. All rights reserved.

eywords: Web panel; Question behavior effect; Online market research; Quasi-experiment; Difference-in-differences estimation; Survey research

ig(nasi

piptci

Many retailers looking for faster and lower-cost ways to con-uct marketing research are building and utilizing proprietaryeb-based panels of their own customers to conduct surveys

nd to obtain feedback on a regular basis (Baker et al. 2010;allegaro and DiSogra 2008; Goritz 2004; Hamilton, Vriens,nd Tramp 2008; Neslin et al. 2009; Sullivan 2012). When form-ng a proprietary web panel, the firm usually selects and invites

representative group of its customers to join the panel. Oncehey join, customers are invited from time to time to partici-ate in online surveys regarding various firm-related issues, andlso to provide other types of feedback such as participating innline focus groups (Brüggen and Dholakia 2009; Hamilton,riens, and Tramp 2008). In exchange, they receive incentivesr rewards for participation (Baker et al. 2010).

Web panels offer retailers an effective means to alleviateoncerns regarding privacy, consumer reluctance, and competi-ion with other researchers. Additionally, this movement toward

∗ Corresponding author.E-mail addresses: [email protected] (B.J. Allen), [email protected]

U.M. Dholakia), [email protected] (S. Basuroy).

oip

obrm

ttp://dx.doi.org/10.1016/j.jretai.2015.12.003022-4359/© 2015 New York University. Published by Elsevier Inc. All rights reserv

nteractive online customer research aligns with the strategicoal of retailers to improve online relationships with customersBart et al. 2005). In recent years, many retailers and retail part-ers such as Walmart, Unilever, Victoria’s Secret, Kraft, Procternd Gamble, and Nike have utilized proprietary web panels andimilar customer-feedback groups to gain important consumernput (Baker et al. 2010; Communispace 2015; Passenger 2015).

The purpose of this study is to investigate the effects that webanel participation has on customer purchasing behavior. Specif-cally we ask: Does participating in a retailer-sponsored webanel impact the way that customers shop at the retailer? Givenhe unique aspects of the web panel that we analyze, the keyontribution of this paper is to inform retailers about the surpris-ng side-effects of retailer-sponsored web panels. In this respect,ur work constitutes phenomenon-driven research aimed specif-cally at helping retailers better understand the impact of webanel participation on customer purchasing behavior.

Our results from the field experiment obtained by the usef difference-in-differences analyses indicate four specific
enefits to the retailer from using such web panels. First, ouresults indicate that customers participating in the web panelake more purchases (17 percent more) from the retailer
ed.

http://crossmark.crossref.org/dialog/?doi=10.1016/j.jretai.2015.12.003&domain=pdf

dx.doi.org/10.1016/j.jretai.2015.12.003

mailto:[email protected]



dx.doi.org/10.1016/j.jretai.2015.12.003

1 Reta

apPfDcctasoicwcpncTefikt

somacwo

wpeiwhp

A

(2fietabiipt

rW2

ato(oiaiirultwitp(

nlri“etwttsfific(twlt

vpse2to

TB

48 B.J. Allen et al. / Journal of

fter participation. This result supports the general findings ofarticipants spending incremental “social dollars” (Manchanda,ackard, and Pattabhiramaiah 2015) and participants of satis-action surveys spending more with the firm (Borle et al. 2007;holakia and Morwitz 2002). Second, our results indicate that

ustomers participating in the retailer’s web panel increase theirross-buying (buying from multiple product categories) fromhe firm after participation (an additional 14 percent). This is

new result not documented by prior research and specificallyhows the benefits that can accrue to a retailer from the creationf such web panels. Third, our results indicate that participatingn web panels increases the profit contribution of participatingustomers by about 36 percent. This is a novel finding ase demonstrate that engaging with firms not only causes

ustomers to increase their purchasing but also to increase theirrofitability. Fourth, we find a positive relationship between theumber of surveys completed by the web panelists and theirhange in the number of purchases and amount of cross-buying.his is a new result as prior studies on the question behaviorffect utilizing field experiments, to our knowledge, are con-ned to the administration of one-time surveys. None, to ournowledge, have examined incremental effects of respondingo multiple consecutive surveys on customer behavior.

The rest of the paper is organized as follows. In the nextection, drawing upon the findings of extant research, we developur research hypotheses. After that, we describe the dataset andodeling methodology used to test the hypotheses, followed by

presentation of the model results and robustness checks. Weonclude the paper with a discussion of the contributions of ourork, its managerial implications, and limitations, also pointingut future research opportunities.

Conceptual Overview and Research Hypotheses

For ease of exposition and to align hypotheses developmentith subsequent empirical tests, the discussion in this section isresented in two parts. The first part develops the first hypoth-sis about changes in customer buying behaviors and theirmpact on the retailer from participation in the retailer-sponsoredeb panel. The second part explores and develops the secondypothesis about the effects of number of surveys completed onanelists’ behaviors.

Summary of Question-Behavior Effects

Research on the so-called “question behavior effect”Dholakia 2010; Gollwitzer and Oettingen 2008; Sprott et al.006) has demonstrated that when customers participate in arm-sponsored purchase intention or satisfaction survey, theyxhibit more purchase loyalty toward the firm. Despite the facthat the current study is phenomenon-driven research aimedt better understanding and helping the field of retailing, weelieve the phenomenon we document can be explained primar-
ly through three potential mechanisms that have been uncoveredn the question-behavior effect literature as resulting from surveyarticipation: increased attitude accessibility, increased posi-ivity of attitudes, and greater functional knowledge about the
pp

iling 92 (2, 2016) 147–161

etailer’s offerings (Dholakia 2010; Dholakia, Morwitz, andestbrook 2004; Morwitz and Fitzsimons 2004; Sprott et al.

006).Considering attitude accessibility first, joining the web panel

nd responding to various surveys posed by the firm will makehe firm more accessible to the consumer, so that its likelihoodf being considered on the next shopping occasion is greaterMorwitz and Fitzsimons 2004). This mechanism for the effectf web panel participation can be traced to self-generated valid-ty theory (Feldman and Lynch 1988) which postulates thatnswering questions about infrequently considered issues resultsn thoughtful cognitive processing in the respondent, and dur-ng this process the response is constructed. Once given, theesponse remains accessible and affects behavior. This is partic-larly relevant in the instance of a web panel where customers areikely to have positive accessible attitudes evident by the fact thathey volunteered to join the panel and help the retailer. Consistentith this mechanism, there is evidence that consumers engage

n mental simulation of target behavior (such as going throughhe buying process at the retailer’s website) through automaticrocessing when they are asked to respond to survey questionsJaniszewski and Chandon 2007; Levav and Fitzsimons 2006).

Second, an invitation to join the web panel may itself sig-al that the retailer is responsive and cares about its customerseading to more positive attitudes toward it. Specifically, prioresearch has shown that participation in the firm’s survey leadsts customers to make positive inferences about it such asThis retailer values my opinions,” “It is making a bona fideffort to please me,” “It is caring and concerned about its cus-omers in general,” “It must be a good retailer” and so forth,hich influences the individual’s behaviors. Note that unlike

he first explanation which relies on increased accessibility ofhe customer’s existing or constructed attitudes, this mechanismuggests that by asking customers to join the web panel, therm is changing their assessment of the firm for the better. Thisnding is consistent with research studying the involvement ofustomers in the new product development process by firmse.g., Fuchs, Prandelli, and Schreier 2010) which has shownhat customers who help the firm in their decision-making abouthich new products to offer develop stronger levels of psycho-

ogical ownership of the firm’s products which in turn increasesheir willingness to pay for them.

Finally, the third mechanism for the positive effects of sur-ey participation is that answering survey questions may simplyrovide new information about different products that the retailerells raising the possibility that panelists will buy from more cat-gories because of this new knowledge (Dholakia and Morwitz002). Together, we expect these effects to manifest in substan-ial economic benefits to retailers from web panel participationf their customers.

he Impact of Retail Web Panel Participation on Customerehaviors

This discussion about the mechanisms through which surveyarticipation affects customers is relevant for how retailer webanel participation may affect behavior of panelists.

Reta

atOvt2(pfihriaiag6ufifpsmm

epcrtaewc

dftbt2ratcpCctobafipp

tmmcnsbt

Hpcp

I

pIoitbcif

trtbQtbitmet

vattsptfic

tc

B.J. Allen et al. / Journal of

When a retailer solicits input through the web panel, the veryct of asking is informative to customers, creating a social rela-ionship between the customer and the firm (Liu and Gal 2011;fir, Simonson, and Yoon 2009). One type of information con-eyed by the request is that the retailer values what the customerhinks (Borle et al. 2007; Dholakia, Morwitz, and Westbrook004), raising perceptions of the retailer’s customer orientationFuchs and Schreier 2011). Such an explanation is also sup-orted by studies showing that customers who were told thatrms solicited customer feedback in creating new products hadigher purchase intentions, willingness to pay, and willingness toecommend the firm (Schreier, Fuchs, and Dahl 2012). It is alson line with studies which have shown that when customers aresked for advice, they feel closer to the firm and show increasesn subsequent propensities to transact and engage with it (Liund Gal 2011). Finally, practitioner surveys also provide conver-ent evidence. In a recent survey of 1,200 Swedish consumers,2 percent indicated greater likelihood of buying a brand’s prod-ct if that brand had sought out their feedback and 56 percentelt more loyal to a brand that made the effort to solicit theirnsights (Sullivan 2012). We expect a similar process to unfoldor those who join the firm’s web panel and participate in theanel’s surveys. Specifically, we hypothesize a positive relation-hip between web panel participation and number of purchasesade by the customer, such that web panelists will purchaseore from the retailer after joining the web panel.Furthermore, because of this hypothesized positive

ffect—more purchasing from the firm, we expect that webanel participation will also increase the dollar amount thatustomers spend with the retailer positively affecting theetailer’s sales revenue from these customers and thus leadingo a greater profit contribution to the retailer. In other words,long with increasing the number of purchases made, we alsoxpect participating in web panels to increase the dollars spentith the retailer which will increase the customer’s profit

ontribution to the retailer hosting the web panel.A second important measure of customer behavior is the

egree to which customers shop multiple product categoriesrom the firm. Purchasing from different categories, referredo as “cross-buying” in the retailing literature, is considered toe an important input into the strength and value of the cus-omer’s relationship with the firm (Kumar, George, and Pancras008). Cross-buying is an important characteristic of valuableetail customers as cross-buying signals the customers’ loyaltycross various categories. Prior research indicates that the cus-omer’s relational attitudes toward a firm are positively related toross-buying (Verhoef, Franses, and Hoekstra 2002), and such arediction is also aligned with QBE research (Borle et al. 2007;handon, Morwitz, and Reinartz 2004). Furthermore, as dis-ussed above, participating in the web panel will not only changehe attitude toward the retailer, but also increases the knowledgef the retailer’s offerings. Specifically, informational benefits areound to accrue from the enhanced information participants are
ble to obtain from their interactions in the web panel and fromlling out the surveys. The customers will learn about variousroducts available through completing the surveys. For exam-le, if a customer frequently shops for jewelry at the retailer but
igpd

iling 92 (2, 2016) 147–161 149

akes a survey about the firm’s movie category, the customeright become aware and then interested in shopping the firm’sovie selection. In this example, the survey put the retailer in the

ustomer’s consideration set for movie purchases, a place it wasot before. Thus, we hypothesize that web panel participationhould also have a positive effect on the customer’s cross-buyingehavior at the retailer. The following hypothesis summarizeshese positive effects of survey participation on web panelists:

ypothesis 1. When customers participate in a retailer’s webanel, such participation will increase: (a) the number of pur-hases from the retailer, (b) the retailer’s total dollar sales, (c)rofit earned, and (d) cross-buying.

mpact of Frequency of Survey Participation

In the first hypothesis, we considered the effects of web panelarticipation on purchasing behaviors of the retailer’s customers.n this section, we consider the potential influence of frequencyf survey participation on the magnitude of changes in purchas-ng behaviors resulting from web panel participation. While theheoretical underpinnings for H1 are similar to H2, it woulde useful to directly address participation frequency and theonceptual rationale for why frequency of participation mightncrease the customer’s purchasing behavior as participationrequency has received less attention in the QBE literature.

Research streams on both the QBE and customer participa-ion in new product development suggest that customers willaise their perceptions of the firm as a result of web panel par-icipation, which will subsequently increase their purchasingehaviors. The most favored mechanism for the occurrence ofBE effects is increased attitude accessibility due to survey par-

icipation (Dholakia 2010). In the prior literature, attention haseen limited to increased attitude accessibility from a singlenstance of survey taking by the customer. It seems reasonablehat if one survey increases accessibility of the retailer, takingultiple surveys over a period of time should strengthen the

ffect and encourage its continuance over a longer time period,hus leading to greater degree of purchasing and cross-buying.

Additionally, there is a relational aspect of soliciting and pro-iding feedback in an exchange relationship between customernd firm. Liu and Gal (2011) found that “soliciting advice tendso have an intimacy effect whereby the individual feels closero the organization, resulting in increases in subsequent propen-ity to transact and engage with the organization (p. 242).” Thisrocess is particularly relevant for this study because an impor-ant mandate of the retailer’s web panel, used as the contextor this study, is to gather advice and opinions regarding how tomprove aspects of the firm’s activities such as its website pages,ommercials, and promotions.

Although Liu and Gal (2011) examined the effects of par-icipating in a single instance of advice giving, their reasoningan be meaningfully extended to multiple instances of provid-
ng feedback. Specifically, we hypothesize that when customersive advice to the firm on multiple occasions such as througharticipation in a larger number of surveys, the intimacy effectiscovered by Liu and Gal (2011) will be enhanced, resulting

1 Reta

ijpsoarttrptrae

cirpappStdwtHpfc

Htc(

T

drsctscfwttar

ie

wptiusistu

igdjrsprit

hfo(ctooi

w1bgciJutaWamsctco


n an increase in customer purchasing behaviors. Such a con-ecture is also supported by the extensive literature in socialsychology on close relationships which views frequency ofocial interactions between people to be a significant aspectf enhancing interpersonal relationships (Berscheid, Snyder,nd Omoto 1989; Milardo, Johnson, and Huston 1983). Thiselational aspect of answering surveys is especially relevant inhe context of firm-sponsored web panels. Whereas conven-ional wisdom suggests that customers in traditional marketingesearch may become annoyed or worn-out as a result of multi-le survey requests, customers in web panels have volunteeredo give feedback to the firm and know they will receive repeatedequests to give feedback. Therefore, customers in web panelsre likely to benefit from the increased contact because they havexhibited a positive attitude toward interacting with the firm.

Furthermore, there is an important practical consequence ofompleting surveys, which is that each survey provides specificnformation regarding products and services offered by theetailer, some of which may be new to panelists. In this sense,articipation in more surveys is incrementally educationalnd informative and can increase buying behavior simply byroviding customers with exposure to more of the retailer’sroducts and services each time (Dholakia and Morwitz 2002).uch an educational effect of survey participation is likely

o be additive with each survey that is completed. All of thisiscussion consistently suggests that completing more surveysill increase the purchase behavior of web panelists and lead

o greater sales revenue and profits from the retailers. While1 proposes that web panel participation will change customerurchasing behavior, H2 suggests that increased participationrequency will lead to larger purchasing changes. Therefore,orresponding to the first hypothesis, we hypothesize:

ypothesis 2. There will be a positive relationship betweenhe number of surveys completed by the web panelist and theustomer’s change in (a) number of purchases, (b) dollar sales,c) profit, and (d) cross-buying.

Overview of Dataset, Model Methodology, and Results

he Dataset

We test the research hypotheses with a quasi-experimentalataset obtained through collaboration with a major onlineetailer. The company is a large online retailer, sells products andervices in dozens of categories, and maintains a web panel of itsustomers to support its marketing research activities. Throughhis exclusively online panel, customers participate in the firm’surveys from time to time and chat occasionally with theompany’s marketing researchers in an accompanying onlineorum. The web panel is maintained on a third party websitehere members log on to participate. When customers log on,

hey see a simplistic interface giving them the option to par-icipate in current activities such as taking available survey(s)nd participating in online forum chats. All departments of theetailer are encouraged to use the web panel to gather customer

aoFi

iling 92 (2, 2016) 147–161

nformation; they must coordinate their projects through themployees who act as the panel’s moderators.

The surveys offered during the study’s timeframe covered aide variety of topics. Examples include surveys about specificroduct lines (e.g., jewelry, house décor, and furniture), pre-esting of advertisements, and perceptions of mobile device usen retail shopping. The majority of survey questions were askedsing Likert scales; there were a few open-ended questions inome of the surveys. The online forums were conversationaln nature and included discussions about favorite brands, pasthopping experiences, and commercials. Our conversations withhe retailer suggested that the surveys were the most valued andtilized output from the web panel.

To create its web panel, the firm invited a random subset ofts existing customers to join. An email was sent to a customerroup that was randomly selected from the retailer’s customerata warehouse which included all customers. Those invitedoined the panel on a first-come, first-served basis. Membershipemained open until the web panel had reached a pre-determinedize. Participation in web panel activities (responding to a survey,articipating in a chat, etc.) was on a voluntary basis. Panelistseceived financial rewards through monthly contests and draw-ngs for store credit (i.e., gift cards). We discuss controlling forhe financial incentives received by customers in a later section.

The web panel examined in this study began in July 2012. Weave data on the panelists’ participation behaviors in the panelor the period from July 2012 to March 2013; the panel was stilln-going at this time. Our dataset consists of all the panelists1,670) who were members of the panel at the time of dataollection. During this time, they were invited to participate in aotal of 27 different surveys conducted by the firm (or an averagef three surveys per month). For each panelist, we have datan the number of surveys completed and detailed transactionalnformation regarding purchase behaviors.

Along with data on web panelists, the firm also provided usith transactional data for the same period for an additional3,504 of its randomly selected customers who were not mem-ers of the firm’s web panel. These customers serve as a controlroup in our analysis. For each customer in the panelist andontrol groups, the available transactional data consist of eachndividual purchase made by the customer during the time periodanuary 2010 to March 2013, along with the date and the prod-ct category from which the purchase was made. Additionally,he firm provided us with the total number of purchases, sales,nd profit over the life of each customer prior to January 2010.e use these as control variables. The firm calculates profit on

per order basis as follows: sales minus cost of goods sold,inus variable marketing costs. Variable marketing costs is the

um of all variable marketing costs associated with bringing theustomer to the website along with any price-reduction given tohem to encourage purchasing. Examples of variable marketingosts include pay-per-click costs (such as those incurred for paidnline search or display ads) and costs of price promotions such
s coupons, discounts, and so forth. Variable marketing costs arenly those costs that can be linked to a particular transaction.or example, TV and magazine advertising costs would not be
ncluded here.

Reta

(dtecoocF

FcBmnwpgt(Twtraafmopjtt

M

tdp2etveatPsb

otpei

evttrepshaatc(

fttttdmomaPatn1wmii

baibicw

D

T

purchases, total sales, total profit, or total product categories)for customer i in time t (pre-period or the post-period) in groupj (panel or control). Time represents a dummy variable of 0 or


Since we have panelist data from July 2012 to March 2013referred to as “post-period” henceforth), we utilize comparableata for the same duration, July 2011 to March 2012 (referredo as the “pre-period” henceforth) from the previous year toxamine effects of participating in the retailer’s web panel onustomer behavior. The equivalent time period from the previ-us year allows for a fair comparison given the seasonal naturef the retailer’s business. We note that such a year-over-yearomparison is routine in retail analytics (e.g., National Retailederation 2013; O’Donnell 2013).

The following unique strengths of our dataset are noteworthy.irst, the retailer is an exclusively online retailer, and requiresustomers to sign into their account before making any purchase.ecause of this constraint, we are able to observe every purchaseade by customers in our dataset, ensuring that our analysis is

ot confounded by any untracked or missed purchases. Second,e have the panelists’ transactional data before and during webanel membership, along with transactional data for a controlroup, which enables us to study the effects of web panel par-icipation using a difference-in-differences estimation approachBertrand, Duflo, and Mullainathan 2004; Wooldridge 2009).hird, we have purchase data before the analysis period so thate can control for individual purchasing behavior as well as

rends prior to the pre-panel period, allowing us to check forobustness of our results. Fourth, because the web panel is oper-ted separately from the company’s website, and is hosted by

third-party marketing research firm, our results are not con-ounded by the possibility that panelists could be transactingore simply because of visiting the retailer’s website more often

n account of logging on to the panel. Fifth, virtually all webanelists joined at approximately the same time (99.8 percentoined within the first month), which helps to alleviate concernshat the results obtained may be different based on time spent inhe panel.

odel and Estimation for Hypothesis 1

To test our hypothesis regarding the effects of web panel par-icipation, we model the data as a quasi-experiment and employ aifference-in-differences (hereafter DD) estimator to assess webanel participation effects (Bertrand, Duflo, and Mullainathan004; Wooldridge 2009). This econometric estimation approachmploys a pre-period versus post-period and tests versus con-rol methodology, which allows us to control for, and rule out,arious alternative explanations such as the possibility that pan-lists are just systematically different from the control groups the reason for the modeled relationship between panel par-icipation and change in behavior (Manchanda, Packard, andattabhiramaiah 2015). By adding a pre-period (as opposed totudying the post-period alone) we model the same customerefore (pre) and during (post) web panel participation.

A key advantage of using the DD estimator is that regardlessf whether panelists purchase more or less when compared to
he control group at the aggregate level in the pre-period orost-period, the model remains unbiased because the casualstimation is not the volume of purchases, but rather the changen purchasing behavior from the pre to the post-period. For
pf

iling 92 (2, 2016) 147–161 151

xample, if the number of purchases is correlated with the groupariable before the quasi-experiment even begins (pre-period),he group difference estimator for the pre-period (and possiblyhe post-period) would be biased, but the DD estimator stillemains unbiased (Stock and Watson 2007). Additionally, bymploying a control group, we control for differences from there-period to the post-period that may have occurred due to rea-ons other than participation in the web panel. Such an approachelps alleviate concerns of endogeneity or reverse causality.1 It islso important to note that aggregating the customer-level trans-ctional data into two time periods (rather than utilizing a moreime-series approach) mitigates the possibility of potential serialorrelation and grouped error term effects in our DD estimationsee Bertrand, Duflo, and Mullainathan 2004 for details).

We employed two additional criteria in selecting customersor inclusion in the treatment and control groups. First, cus-omers had to have purchased at least once from the firm prioro the beginning of the pre-period. This requirement ensureshat customers in the analysis did not first become aware ofhe company (or consider purchasing from them) in the mid-le of the pre or post period. Second, customers had to haveade at least one purchase during the nine-month pre-period and

ne purchase during the nine-month post-period. This require-ent ensures that customers in the panelist and control groups

re the firm’s active customers (Baker et al. 2010; Manchanda,ackard, and Pattabhiramaiah 2015). These selection criteriare important because different entry and exit times betweenhe customer groups could pose a threat to the assumption ofo sample composition changes (Blundell, Duncan, and Meghir998; Manchanda, Packard, and Pattabhiramaiah 2015). Whilee believe this to be the most rigorous and appropriate selectionethod, in the robustness check section we also report results

ncluding the entire sample of control group and panel membersn our dataset.

Additionally, we deleted one outlier from the panel groupecause its pre-period number of orders was 34 standard devi-tions above the mean; this type of outlier deletion is commonn the retailer’s analyses because the customer may actuallye a small business owner (e.g., drop-shipper) rather than anndividual consumer. 492 members in the panel group and 429ustomers in the control group satisfied the above criteria andere included in the study (see Table 1 for summary statistics).The specific model employed to test our hypotheses via the

D estimator can be expressed as:

ransactionijt = α + β1Timet + β2Panelj + β3Timet ∗ Panelj

+ β4PastBuyingij + εijt (1)

where Transaction is the transactional variable of interest (total

1 There is a possibility that due to the trends going into the experiment thatanel membership is, in itself, an endogenous variable. This issue is discussedurther in the Robustness Checks section.

152 B.J. Allen et al. / Journal of Reta

Table 1Summary statistics for web panel and control groups.

N Mean SD

Number of purchasesPre-period

Panel 492 3.01 2.64Control 429 2.32 2.24

Post-periodPanel 492 3.23 3.00Control 429 1.99 1.90

SalesPre-period

Panel 492 326.46 499.08Control 429 292.24 459.47


ProfitPre-period

Panel 492 34.47 114.62Control 429 48.00 96.43


Cross-buyingPre-period

Panel 492 3.00 2.42Control 429 2.28 2.01

Post-periodPanel 492 3.34 2.70

1rtt0idav

o

(

wPcfptc

diad

t(tfit(fieofbFpWinftowicri

R

DarvtWcttoiia(ibpp

In stating H1b, we predicted that sales revenue would increasedue to participation in the web panel. Results indicate that theDD estimator is in the hypothesized direction; however we

2 It is likely that the different dependent variables are related across the cus-tomers in the dataset. Therefore, we ran a multivariate analysis of variance

Control 429 2.14 2.12

if the transaction occurred in the pre-period or post-period,especitvely. β1 captures any overall transactional differenceshat are common between the panel and control group acrosshe time periods. Panel is a dummy variable that takes the value

if the customer is in the control group and 1 if the customers in the panel. Therefore, β2 captures any overall transactionalifferences between the panel and control group. β3 is the inter-ction between the Panel and Time variables and represents theariable of interest or the DD estimator.

The full calculation for β3 can be represented (using numberf purchases as an example) as:

PurchasesPost,Panel − PurchasesPre,Panel)

− (PurchasesPost,Control − PurchasesPre,Control) (2)

hile controlling for other variables in the model, where Post orre is the period, and Panel or Control represents the group. Thisalculation essentially uses the change in number of purchasesor the control group to estimate the “predicted” change for theanel group. The DD estimator becomes the difference betweenhe observed change for the treatment group and the “predicted”hange for the treatment group.

For tests of hypotheses 1a–d, the number of purchases isefined as the total number of separate purchases made by the
th customer in the time period. Sales and profit are the total salesnd profit, in dollars, associated with the ith customer’s ordersuring the measured timeframe, respectively. In other words,
u(tl

iling 92 (2, 2016) 147–161

hey are the total sales and profit for each customer during time tthe nine-month timeframe). Cross-buying is operationalized ashe number of separate product categories a customer purchasedrom during the specified time period from the retailer’s approx-mately one hundred different categories in its product-levelaxonomy. PastBuying is included to control for the customer’sboth panelists and control group) previous relationship with therm as it pertains to that specific transactional variable of inter-st. PastBuying is operationalized as the total for that variablever the life of the customer before the pre-period. Specifically,or number of purchases, it is operationalized as the total num-er of purchases made by the customer before the pre-period.or sales and profit it is operationalized as the total sales androfit, respectively, made by the customer before the pre-period.hen cross-buying is the dependent variable, it is operational-

zed as total number of purchases (similar to PastBuying whenumber of purchases is the DV). We use this operationalizationor cross-buying because we do not have cross-buying informa-ion over the entire life of the customer, and cross-buying andrders are highly correlated. Nevertheless, as a robustness check,e ran Eq. (1) for cross-buying without PastBuying included

n the model and the results are very similar. The t-values ofoefficients for all models are estimated using heteroskedasticityobust standard errors. See Table 2 for a summary of the variablesn the model and explanation of the variable operationalization.

esults: Hypotheses 1a–d

To test H1a–d, we ran four linear regression models withD-estimators employing the number of purchases, sales, profit,

nd number of separate product categories (cross-buying) as theespective dependent variables (see Eq. (1)).2 The results are pro-ided in Table 3. The model for number of purchases shows thathe DD estimator (β = .548, p < .01) is positive and significant.

hen compared to the change in number of purchases for theontrol group, purchases of panelists grew from the pre-periodo the post-period after joining the web panel, providing supporto Hypothesis 1a. The estimate of .548 is the incremental numberf purchases above and beyond that which would have occurredn the absence of panel membership. In order to estimate thencremental impact as a percentage, we used the post-periodverage number of purchases for the web panelists as the basesee Table 1). Using this approach, the estimated incrementalmpact due to panel participation is 17 percent (.548/3.23; 3.23eing the average number of orders for panelists in the post-eriod). In other words, web panel participation accounts for 17ercent of the customer’s post-period number of purchases.

sing the four transactional variables as dependent variables and the two groupscontrol vs. panel) as the independent variable. The results demonstrate thathere are indeed differences in the groups worth further examination (Wilks’ambda = .9868, p = .016).

B.J. Allen et al. / Journal of Retailing 92 (2, 2016) 147–161 153

Table 2Summary of variables and operationalization.

Variable Explanation

# of Purchases Number of separate purchases (orders) made by the customer in the time period.Sales The total dollar value of all purchases made in the time period.Profit The total profit (in dollars) of all purchases made in the time period. Profit is calculated on a per order basis as

such: Sales minus costs of goods sold minus variable marketing costs. Variable marketing costs is the sum of allmarketing costs associated with bringing the customer to the website and enticing them to order. For example,variable marketing costs include pay-per-click costs (such as paid online search or display ads) and coupon costs.Variable marketing costs are only those costs that can be linked to a particular transaction. For example, TV andmagazine advertising costs would not be included here.

Cross-Buying The total number of unique product categories the customer purchased from in the time period. As a reference,the retailer utilizes about 100 different categories in its product-level taxonomy.

Time Indicates whether the dependent variable is measuring the transaction in the pre or post period. The variabletakes the value of 1 if it occurred in the post-period and 0 if it appeared in the pre-period.

Panel Indicates whether the customer is a member of the web panel. The variable takes the value of 1 if the customer isa web panel member and 0 if the customer is in the control group.

PastBuying Operationalized as the total, for the specific transactional variable, over the life of the customer before thepre-period. For number of purchases, it is operationalized as the total number of purchases made over the life ofthe customer. For sales, it is the total dollars sales over the life of the customer. For profit, it is operationalized asthe total profit over the life of the customer. For cross-buying, it is operationalized as the total number ofpurchases over the life of the customer (similar to PastBuying for number of purchases).

Additional variables in Eq. (3)a

Surveys Number of surveys the panelists completed over the course of the nine-month period.LTP (likelihood to purchase) A self-reported likelihood to purchase question worded: “I am likely to make my next online purchase from

[firm].” Panelists provided their responses on a 1–5 Likert scale, anchored by “strongly disagree” and “stronglyagree.”

FinIncent A dummy variable that takes the value of 1 if the customer had received a financial incentive from participatingin the panel and 0 if not.

Age Age of the panelist, measured in years.Gender A dummy variable that equals 1 if the panelists is a male, 0 if a female.Income The retailer only collected income as a categorical variable grouping income into eight different income

e dum

y are

czpba

cp

TD

FR

ECTPDPO

NtpN

*

H

categories. Therefore, we includ

a These additional variables were not used in Eq. (1) (DD model) because the

annot conclude that the estimator is significantly different fromero (β = 55.91, p = .22). One possible reason that number of
urchases is significant but the sales revenue is not could beased on the nature of the broad-line retailer. This retailer sells
wide assortment with a large variation in selling prices. This

H

c

able 3ifference-in-differences model results for tests of H1.

Number of purchases Sales

-Stat F(4, 1837) = 41.67*** F(4, 1837) = 6.82 .108 .138

stimatesonstant 2.073 (.105)*** 218.324 (29.209ime −.329 (.137)* −11.600 (31.97anel .283 (.164)+ −43.870 (39.25D Estimator .548 (.212)** 55.910 (45.709)astBuying .053 (.006)*** .109 (.032)***

bservations 1,842 1,842

ote: Dependent variable is number of purchases, total sales, total profit, or number ootal orders before the pre-period for number of purchases and cross-buying. It is oprofit, respectively.

= 1,842 (921 customers times two time periods).* p < .05.

** p < .01.** p < .001.+ p < .1.eteroskedasticity robust standard errors shown in parentheses.

my variables to represent the different income divisions.

only available for the panel customers and not the control group.

haracteristic could lead to large fluctuations in average saleser order, which may cause the standard errors to be high. Thus
1b is not supported.Considering H1c next, which hypothesized that profit per
ustomer would increase, the results provide support. The DD

Profit Cross-buying

7*** F(4, 1837) = 3.45** F(4, 1837) = 34.11***

.094 .102

)*** 35.712 (6.051)*** 2.045 (.098)***

5) −4.513 (6.122) −.133 (.138)1) −18.783 (7.458)* .347 (.144)*

16.871 (8.735)+ .464 (.209)*

.112 (.039)** .050 (.006)***

1,842 1,842

f separate product categories (cross-buying). PastBuying is operationalized aserationalized as total sales and total profit before the pre-period for sales and

1 Reta

ecicwcTa

lmT(fcbbope

FNp

ae

R

o(cna

atatt


stimator (β = 16.87, p = .054) is marginally significant and indi-ates that compared to the control group, profit per customerncreased by $16.87 for the panelists. Using the total profit perustomer in the post-period as the base ($46.83, see Table 1),e conclude that web panel participation accounted for 36 per-

ent (16.87/46.83) of post-period profitability for web panelists.hus panelists not only increase their number of purchases, butre also more profitable to the firm.

In stating H1d, we predicted that joining the web panel willead to an increase in customers’ cross-buying, and again, the

odel estimates (presented in Table 3) support this hypothesis.he DD estimator for cross-buying is positive and significantβ = .46, p < .05). Web panelists increased the number of dif-erent categories they purchased from by an average of .46ategories after joining the web panel. Using the average cross-uying for panelists in the post-period (3.34 categories) as thease, web panel participation accounts for 14 percent (.46/3.34)
f the product categories purchased from the retailer during theost-period. Fig. 1 provides a graphical illustration of the DDstimator: the gap between the actual numbers in the post-period
0

0.5

1

1.5

2

2.5

3

3.5

Post-period PurchasesPre-period Purchases

Actual Control

Actual Panel

Predicted Panel

$0

$10

$20

$30

$40

$50

$60

Post-period ProfitPre-period Profit

Actual Control

Actual Panel

Predicted Panel

0

0.5

1

1.5

2

2.5

3

3.5

4

Post-period Cross-BuyingPre-period Cross-Buying

Actual Control

Actual Panel

Predicted Panel

ig. 1. Graphical illustration of the difference-in-differences (DD) estimator.ote: The gap between the actual and the predicted transactional behavior in theost-period for the panel represents the DD estimator.

pato

arm(srp2ipgtwuontpodsmp

at(fiwbaOfpp

iling 92 (2, 2016) 147–161

nd the predicted numbers in the post-period represents the DDstimator.

obustness Checks

As discussed earlier, our DD-based methodology is rigor-us and appropriate for the research questions being examinedManchanda, Packard, and Pattabhiramaiah 2015). Even so, weomprehensively considered and addressed its potential weak-esses to rule out alternative explanations as best as we couldnd to assess the robustness of our findings.

Dealing with self-selection. One alternative possibility toccount for the findings is that customers who self-selected intohe web panel are different from the control group in their over-ll buying relationship with the retailer, which in turn driveshe pre/post change (i.e., increase) in buyer behavior. In ordero rule out this alternative explanation, we employed the kernelropensity score matching methodology (Heckman, Ichimura,nd Todd 1998) to assess the robustness of the DD estima-ion for number of purchases, profit, and cross-buying, the threeutcomes for which differences were significant in the analysis.

While complete randomization of the treatment group islways optimal, it is often difficult to execute in field studies ineal settings and such was the case here. DD estimation is com-only applied even in the absence of complete randomization

e.g., Autor 2003; Bronnenberg, Dube, and Mela 2010). Propen-ity score matching is often utilized to determine robustness ofesults obtained from DD estimation in the absence of com-lete randomization (Manchanda, Packard, and Pattabhiramaiah015). The propensity score matching method estimates thempact of a treatment by matching customers from the webanel group (treated person) with customers from the controlroup (untreated person) who are similar based on characteris-ics observed before the measurement period. In our analysis,e matched customers based on a propensity score calculatedsing the past buying behavior for the transactional variable(s)f interest before the measurement period (PastBuying). Usingumber of purchases as an example, we compared control cus-omers whose previous total past purchases were similar to theanelists. While we would have liked to match customers basedn other observable variables such as demographics, the retailerid not capture demographic information for the control group;o it was not available to us. Nevertheless, we propose that theain self-selection variable of interest is past (before the study

eriod) buying relationships of the customer with the firm.The results of the kernel matching estimation closely

pproximate the impact of web panel participation found inhe DD analysis. For number of purchases, it is .945 ordersp < .05), or 29 percent (.945/3.23) of post-period orders. Thisnding indicates that if our original results are biased in anyay, they are actually biased toward greater conservatismecause the DD estimator using the kernel matching revealedn even greater difference than the DD estimate obtained using
LS. Additionally, the DD estimator using kernel matching
or profit is $19.658 (accounting for 42 percent of post-periodrofitability) and for cross-buying is .591 (accounting for 18ercent of post-period cross-buying), both of which are close


Table 4Difference-in-differences estimation using propensity score matching.

Originalestimation

Kernelmatching

Nearestneighbor

Orders: DD .548 .945 1.061Profit: DD $16.871 $19.658 $30.536Cross-buying: DD .464 .591 .688

Ns

ts

s(tmiorsce

Astwgsr(2

tn(tpE22mppamit

sphttaT

Table 5Difference-in-differences estimation relaxing inclusion requirements.

Originalestimation

Using allcustomers

Orders: DD .548 .377Profit: DD $16.871 $12.84Cross-buying: DD .464 .373

Observations 1,842 30,346a

N

fpptwtigmst

mfp(wamaiottoobTitwacmwctmi

r

ote: All propensity score estimates are significant at p < .05 using bootstrappedtandard errors.

o our original estimation. All three estimates are statisticallyignificant at p < .05 using bootstrapped standard errors.

Additionally, we also employed a different form of propensitycore matching using the “nearest neighbor matching” methodHeckman, Ichimura, and Todd 1998). Using this approach,he results support the original DD estimator and the kernel

atching results. The nearest neighbor approach estimated thencremental impact of web panel membership on number ofrders, profit, and cross-buying to be 1.061, $30.536, and .688,espectively (all significant at p < .05). These results, again, con-istently indicate that if at all, the original estimates err on theonservative side. Table 4 provides a comparison of all DDstimates obtained using the different methods.

Assessing existing trends before the measurement period.nother threat to the interpretation of our results is from the pos-

ibility that panel membership is not a causal factor. Specifically,he results may be subject to reverse causality if customers whoere already increasing their shopping with the online retaileroing into the measurement period were more likely to self-elect into the web panel. This is a particular concern oftenaised about respondents of survey-based marketing researchDholakia 2010; Manchanda, Packard, and Pattabhiramaiah015; Sprott et al. 2006).

To rule out endogeneity and reverse causality in DD estima-ion, we must show support for the assumption that there waso trend in buying behavior before panel membership beganAtanassov 2013; Bertrand and Mullainathan 2003). To assesshis issue, we tested for any extant trend difference between theanel and control group going into the pre-period. We employedq. (1) and using a DD estimator examined the pre-period (July011 to March 2012) against the prior nine-month period (July010 to March 2011) referred to as the “pre-pre period”. In thisodel, β3 is our DD estimator and PastBuying represents the

ast buying behavior for the transactional variable before there-pre period. This approach uses the same test versus controlnd pre versus post-period methodology to estimate a DD esti-ator: if the DD coefficient is found to be significant, then there

s a possibility that our analysis is biased because a differencerend was observed even before our quasi-experiment started.

In selecting the panel group and control groups, we used aimilar methodology as the one used for the original model. Inarticular, to be included in the robustness analysis, customersad to have ordered at least once each in the pre-period and
he pre-pre-period, and must have ordered at least once beforehe pre-pre-period began. This ensures the same DD estimationssumptions discussed previously are retained for this model.he results of this model revealed a nonsignificant DD estimator
bSpn

ote: Estimations using all customers are significant at p = .000.a N = 30,346 (15,173 customers times two time periods).

or number of purchases (β = .038, p = .90), profit (β = 7.633, = .70), and cross-buying (β = −.296, p = .40). These resultsrovide some confidence against pre-existing trends, indicatinghat our results cannot be accounted for by the explanation thateb panelists were exhibiting a different purchasing trend than

he control group going into the test period. As is illustratedn Fig. 1, a key assumption of DD estimation is that the tworoups would have exhibited similar slopes in the absence of theodeled event; the results of the robustness check demonstrate

upport for this assumption by showing that the slopes of thewo groups before the test period were similar.

Replicating the analysis by relaxing inclusion require-ents. In addition to the two robustness checks discussed thus

ar, we also modeled the DD estimator using all panelists (1,669anelists: 1,670 minus the one outlier) and all control customers13,504). We did so to assess the possibility that our resultsere sensitive to the strict inclusion criteria we used to define

ctive customers for our study. While we believe that this is theost methodologically correct procedure given the strict DD

ssumptions, we nevertheless ran our models for all customersn the dataset in order to test the robustness and generalizabilityf our results. Our results indicated that the DD estimator forhe entire dataset is similar to the previously reported results forhe three outcomes for which differences were significant in theriginal analysis. Specifically, the DD estimators for number ofrders (β = .377, p = .000), profit (β = 12.84, p = .000), and cross-uying (β = .373, p = .000) are all positive and significant (seeable 5). We note that since the number of customers included

n this robustness check is different than the original estimation,he means for the groups are also different which, in part, ishy the DD estimators are slightly different. For example, the

verage orders in the post-period for all panel members (1,669ustomers) is 1.69, while the panel group in the original esti-ation (429 customers) is 3.23 (as shown in Table 1). Thus,hile the DD estimator is lower using all panel members (.377

ompared to the original .548), the DD estimator in percentageerms is actually larger, 22 percent (.377/1.69), using all panelembers. Thus, the results in Table 5 are simply to show that

ncluding all customers leads to directionally similar results.Taken together, these additional results show that joining a

etailer-sponsored web panel has significant effects on customerehaviors, and are robust to various alternative explanations.
pecifically, orders, profit, and cross-buying were all affectedositively as a result of participation, while we observed no sig-ificant change for sales. In summary, we find consistent and

1 Reta

rtt

M

niiairmFHc

tpvi(awtpicfdaebrrtioi

tcetwpdmrthitvtd

iaa

e

�

wctrtblit

(ttavomFaaceiod

R

toAemntpsiiv(mp


obust support for H1a, H1c, and H1d. Next, we turn our atten-ion to better understanding the role of survey participation inhe magnitude of changes in behavior of web panelists.

odel and Estimation for Hypothesis 2

Hypothesis 2 predicted a positive relationship between theumber of surveys answered by the panelist and his or her changen buyer behavior. Thus, it posits that the change in buyer behav-or is not the same across all panelists, but will be positivelyssociated with the panelist’s participation level as operational-zed by number of completed surveys. This hypothesis is onlyelevant for panelists, and consequently, we estimated a differentodel (not a DD estimator) using only panel members to test it.urthermore, the majority of variables used in the model to test2 are only available for panel members, thus the control group

ould not be included.In testing H2, we measured the number of surveys as the

otal surveys the panelist participated in during the web paneleriod. Panelists in our sample answered an average of 9.48 sur-eys (SD = 7.98) over the study’s nine-month period. PastBuyings operationalized the same way as in the previous model (Eq.1)). In addition, we included a number of other control vari-bles in our analysis. First, it could be suggested that panelistsho already knew they wanted to purchase in the future joined

he web panel. To rule out this possibility, we capture the webanelists’ self-assessed likelihood to purchase in the future andnclude it as a control variable. A self-reported likelihood to pur-hase was assessed from the panelists’ responses to a questionrom the very first panel survey which asked respondents theegree to which they agreed with the following statement: “Im likely to make my next online purchase from [firm].” Pan-lists provided their responses on a 1–5 Likert scale, anchoredy “strongly disagree” and “strongly agree”. Survey questionsegarding future purchasing intent are often utilized in retailingesearch (e.g., Sirohi, McLaughlin, and Wittink 1998). We notehat we could not use this in the models to test H1a–d because its only available for panelists (and not the control group). Sec-nd, we also included demographic variables (age, gender, andncome) as control variables which were available for panelists.

Third, we included a control for financial incentives offeredo panelists. We noted earlier that the firm conducted monthlyontests and drawings for company credit (i.e., gift cards) inxchange for participating in the web panel, as is common indus-ry practice. Panelists received a financial incentive either byinning a contest via random drawing (each time the panelistarticipated in a survey or forum they received an entry in therawing), or by winning a contest such as best insightful com-ent or highest participation. The web panelists in our data

eceived an average of $3.70 per month in incentives duringhe nine-month time period of the study. In the dataset, aboutalf of the participants (48 percent) had won at least one draw-ng/contest for company credit during the nine-month period
hereby receiving a financial incentive. The number of contestsariable is highly skewed with the majority of observations equalo 0 or 1. For parsimony, and to better capture the non-linearynamics, we create a dummy variable, which takes the value 1
nchr

iling 92 (2, 2016) 147–161

f a customer has ever won a contest, 0 otherwise, and insert it as control. Rinallo and Basuroy (2009) use a similar dichotomypproach when operationalizing highly skewed count data.

The specific linear regression model used to test H2 isxpressed as:

Transactioni = α + β1Surveysi + β2PastBuyingi + β3LTPi

+ β4FinIncenti + β5Demoi + εi (3)

here �Transaction represents the post-period and pre-periodhange for the transactional variable of interest of the ith cus-omer (post-period minus pre-period). The variable Surveysepresents the number of surveys panelist participated in duringhe nine-month period; PastBuying is the past buying behaviorefore the pre-period (same as Eq. (1)); LTP is the self-reportedikelihood to purchase; and FinIncent represents the financialncentive dummy. Table 6 contains the descriptive statistics andhe correlations.

We note the high correlation between Surveys and FinIncentr = .696) that is reported in Table 6. Intuitively, we would expecthese variables to be highly correlated given that the only wayo earn financial incentives is to participate in the web panel,nd one of the major ways of participating was to take sur-eys. We also note that this high correlation does not impedeur modeling procedure as the VIFs for these variables, in allodels, are ≤2.0, indicating that multicollinearity is not an issue.inally, Demo represents a vector of demographic control vari-bles and includes age (in years), gender (female = 0, male = 1),nd income. The retailer from which we received the data onlyollected income as a categorical variable grouping income intoight different income categories. The income categories are notn equal intervals, so in order to deal with the nonlinear naturef the data we include dummy variables to represent the eightifferent income divisions.

esults

To test H2a–d, four separate models were ran to measurehe hypothesized effects of survey participation level on numberf purchases, profitability, sales, and cross-buying, respectively.lthough sales was not significant in the DD model, we nev-

rtheless include it in this second analysis for consistency. Allodels utilized heteroskedasticity robust standard errors. The

umber of surveys variable (Surveys) was significant and inhe hypothesized direction for number of purchases (β = .070,

= .019; see Table 7 for results). This shows a positive relation-hip between number of surveys taken and a positive changen the number of orders placed after controlling for purchasentent, demographics, and financial incentives. Number of sur-eys taken was also marginally significant for cross-buyingβ = .056, p = .072), but it was not statistically significant in theodels predicting the change in sales or profit. These findings

rovide support for H2a and H2d, indicating that the more the
umber of surveys that a panelist completes, the greater is thehange in purchasing and cross-buying after joining the panel;owever, it does not impact the sales revenue or profits theetailer earns from the panelist.


Table 6Descriptive statistics and correlations for model testing H2.

Mean SD Surveys PastBuyinga

(Purchases &Cross Buying)

PastBuyinga

(Profit)PastBuyinga

(Sales)LTP FinIncent Age Gender

Surveys 9.48 7.98 1.00PastBuyinga

(Purchases &Cross Buying)

12.69 15.66 .116 1.00

Past Buyinga

(Profit)152.55 258.54 .046 .508 1.00

PastBuyinga

(Sales)1,391.49 1,960.55 .068 .744 .702 1.00

LTP 3.78 0.95 −.095 −.005 .014 −.018 1.00FinIncent .47 .50 .696 .053 .025 .060 −.049 1.00Age 47.65 12.04 .121 .097 .112 .041 .054 .043 1.00Gender .21 .41 .024 .017 −.071 .035 −.034 −.009 .009 1.00� Number of

Purchases.20 3.18 .222 .046 .017 .046 −.073 .168 .089 −.026

� Sales 47.44 664.40 .072 .032 .046 .185 −.049 .075 .022 .049� Profit 13.79 166.33 .000 .005 −.195 .103 −.023 .011 −.017 −.005� Cross-Buying .33 3.00 .204 .073 .037 .059 −.049 .168 .033 −.022

Note: N = 419; some customer observations are dropped due to incomplete data.ber of

b

biev(β

fitmt

o

ptTsaopp

TP

FR

ECSPLFAGIO

N

*

H

a PastBuying is operationalized as total orders before the pre-period for numefore the pre-period for sales and profit, respectively.

To test the possibility of tapering (an inverted U-relationshipetween number of surveys and dependent measures), wenserted a quadratic term for Surveys in addition to the lin-ar term. The quadratic term is insignificant for all fourariables—number of purchases, cross-buying, sales, and profitβ = .001, p = .698; β = .000, p = .973; β = −.017, p = .979;

= .144, p = .363, respectively). In examining the impact of thenancial incentive variable in the models, results indicated that

his variable was insignificant in all the linear and quadratic
odels, indicating that financial incentives did not predict the
ransactional behaviors of panelists.Together, these results shed light on the rather complex effects

f the extent of retailer’s web panel survey participation on t

able 7anel members’ purchase behavior model results for tests of H2.

Number of purchases Sales

-Stat F(13, 405) = 2.06* F(13, 42 .067 .048

stimatesonstant .084 (1.089) −24.91urveys .070 (.030)* 1.152 (astBuying .004 (.010) .063 (.0TP −.190 (.179) −28.08inIncent .230 (.428) 69.847

ge .017 (.013) .801 (2ender −.271 (.335) 63.036

ncome dummies included Yes Yes

bservations 419 419

ote: N = 419; some customer observations are dropped due to incomplete data.* p < .05.

** p < .01.**p < .001.+ p < .1.eteroskedasticity robust standard errors shown in parentheses.

purchases and cross-buying. It is operationalized as total sales and total profit

anelists. They indicate that purchasing growth is unevenly dis-ributed across panelists based on their actions while in the panel.hese results show the positive relationships between takingurveys and the customers’ change in their number of ordersnd in their cross-buying, but no effects on the firm’s revenuesr profits earned. Our results also indicate that the change inurchasing behavior is not caused by past purchasing, futureurchase intentions, or financial incentives.

General Discussion

To deal with the significant challenges associated with reluc-ant and hard-to-reach consumers and greater competition from

Profit Cross-buying

05) = 0.77 F(13, 405) = 0.88 F(13, 405) = 2.50**

.057 .061

2 (210.757) 25.479 (29.100) .423 (1.008)7.102) .013 (1.090) .056 (.031)+

42) −.149 (.163) .007 (.008)1 (36.329) −2.799 (5.443) −.075 (.158)(108.689) 5.807 (19.280) .387 (.500).763) .130 (.479) .001 (.012)(79.338) −6.484 (18.956) −.177 (.304)

Yes Yes419 419

1 Reta

oeotwkeamwpifwotasebt

C

towtomseQttr

tficcafbTtep

ofilwww

olikw$csmpsst

b(oTevpqmnotmtppb

I

roPooptntttpc

r(tn


ther retailers also seeking customer feedback, many retail-rs have shifted to building and using proprietary web panelsf their customers for conducting marketing research. Despiteheir growing popularity, little is known about how customereb panels affect participant buying behaviors. Drawing uponnowledge on the question behavior effect in the marketing lit-rature, and employing a large quasi-experimental dataset fromn online retailer that was uniquely suited to study this issue,odeled using a difference-in-differences estimation approach,e found that participating in the retailer’s proprietary webanel had a significant impact on customer purchasing behav-ors. Web panelists made more purchases from and purchasedrom more product categories after participating in the retailer’seb panel surveys. Additionally, we show that participants notnly increased their number of purchases but also increasedheir profitability to the retailer. Our results are robust to variouslternative explanations using robustness checks of propensitycore matching and checking existing trends before the quasi-xperiment, as well as by showing that the change in purchasingehavior is not caused by past purchasing, future purchase inten-ions, and the effect of financial incentives.

ontributions to the Literature

Our study documents four significant and specific benefitso the retailer from creating and maintaining web panels. First,ur results indicate that customers participating in the retailer’seb panel make more purchases from the retailer after participa-

ion. This result supports the general findings of participants ofther marketing programs run by firms such as customer com-unities (Manchanda, Packard, and Pattabhiramaiah 2015) and

urvey research participants spending more with the firm (Borlet al. 2007; Dholakia and Morwitz 2002). The extension of theBE literature to online panels provides a significant extension

o prior findings of survey participation that have been limitedo telephone-based surveys and directly answers the call foretailing research in online settings (Grewal and Levy 2007).

Second, our results indicate that customers participating inhe retailer’s web panel increase their cross-buying (productsrom multiple categories) from the firm after participation. Thiss a new result not documented in prior research and specifi-ally shows the benefits that can accrue to a retailer from thereation of customer web panels. This is an important findings extant research has argued cross-buying to be an importantactor in firm-customer relationships. Thus, increasing cross-uying could result in more loyal customers in the long-term.hird, our results show that the profit of the customers who par-

icipated in the web panel increases significantly, a finding thatxtends prior QBE research on single customer surveys to webanels.

Not only does web panel participation increase customers’rder frequency, it also adds to the firm’s bottom-line. While wend significant changes for orders, profit, and cross-buying, the

ack of significance for sales is a bit surprising. While its estimateas in the hypothesized direction, we could not conclude that itas significantly different from zero (p = .22). In our discussionith the retailer, they mentioned that sometimes the broad nature

otaw

iling 92 (2, 2016) 147–161

f their product line (e.g., products with very low costs and veryarge costs) leads to large fluctuations in sales, which can resultn the difficulty of detecting sales changes when measuring mar-eting initiatives. This logic seems consistent with our dataset,here sales of panel members in the post-period has a mean of370.77 and standard deviation of $581.07. Alternatively, onean surmise that participation in web panels does not neces-arily make customers increase their sales volume, but simplyakes the customers’ orders less costly to acquire (which is why

rofit was significant but not sales). But, given that orders wasignificant in our study and that past research in QBE demon-trates that survey participation increases purchasing behavior,his rationale seems less likely.

Fourth, our results indicate that a positive relationshipetween the number of surveys completed by the web panelistsup to a maximum of 27) and the customer’s change in numberf purchases and cross-buying after participating in the panel.his is a novel finding as all studies on the QBE utilizing fieldxperiments, to our knowledge, are confined to one-time sur-eys of customers. Interestingly, we do not find that sales orrofit are related to participation frequency. It appears that fre-uency of participation can predict whether customers will orderore and whether they will order from more categories, but does

ot predict the actual volume they will order or the profitabilityf orders. In the case of profit, we may get this result becausehe profit contribution of an order is highly contingent on theargin of the product category from which the customer orders;

his may make it difficult to detect a linear relationship betweenrofit and participation frequency. Thus, we see that in aggregate,rofit increased due to panel participation, but not necessarilyecause of increased participation frequency.

mplications for Managers

Our paper adds to the literature on QBE effects and is alsoelevant to the newer literature that examines the economicsf online customer communities. As Manchanda, Packard, andattabhiramaiah (2015, p. 384) write: “While there is much the-retical and survey-based research available on the motivationsf consumers who participate in such communities, there is aaucity of research that uses behavioral (market) data to quantifyhe possible economic benefits to firms that set up these commu-ities.” Even though we studied an agglomeration of customershat was limited in scope and time and much of their interac-ion within the panel was with the firm’s surveys, we still findhat participation in it accounted for an additional .548 orderser customer, increased profitability by $16.87, and increasedross-buying by .46 categories over the nine-month study period.

We note that magnitude-wise, our results are in line with theecent findings of Manchanda, Packard, and Pattabhiramaiah2015) who find that online community participants increasedheir purchasing by about 18 percent after joining the commu-ity. Importantly, our findings show that panel participation not
nly increases ordering by customers, but also how profitablehey are to the retailer. This has important implications for man-gers. Our results strongly suggest the possibility that customersho engage with the retailer by participating in the panel are

Reta

mm

nrcrcfikgai

poiIacei2vp

S

rddieawmbdaqmbsrcefitp

wuiibi

coQaa

aitpbwiiatdtmosiritswo

atomwoatpr

nfitdptetc


ore likely to have top-of-mind awareness and require fewerarketing initiatives to continue their relationship.Our overall results indicate that the web panel can yield sig-

ificant economic benefits for the retailer beyond the insightseceived and utilized from the customer feedback. Web-basedustomer research is a dynamic customer interaction betweenetailers and customers and should be thought of in terms of theustomer experience (Grewal, Levy, and Kumar 2009). Thesendings underscore the need for retailers to view online mar-eting research activities beyond a traditional “cost center” toain customer insights, to a customer relationship strengtheningctivity that can yield, as in the present case, a positive economicmpact for the retailer.

Our results regarding the incremental positive effects of com-leting additional surveys, with no tapering up to the maximumf 27 surveys over nine months, also has considerable practicalmportance and contributes substantially to the QBE literature.ts key implication is for quarantine rules that web panel man-gers often set and enforce. Quarantine rules are based on theonventional wisdom that frequent solicitation of web pan-lists is undesirable because it causes negative perceptions andncreases likelihood of dropping out from the panel (Sullivan012). Our respondents answered an average of roughly 9.5 sur-eys, or more than one survey per month, and continued to showositive incremental effects of each survey they completed.

tudy Limitations and Future Research

The strengths of this study are its unique setting, theich dataset, the rigorous methodology utilizing difference-in-ifferences estimation, and the consistency of findings across theependent variables in the aggregate. However, like every study,t also has some limitations. The first limitation is that the quasi-xperimental methodology we employed, the fact that the datare from a single large retailer, and that the proprietary panelas composed, managed and incentivized in a particular wayay place limits on the applicability of our results. Care should

e taken to extrapolate our findings to industries that are starklyifferent, or to proprietary panels that are put together and man-ged in significantly different ways (e.g., by rotating customersuickly or with fewer activities to perform). As one example, itight be difficult for a B2B firm or for a small or medium-sized

usiness with few customers to use proprietary panels or obtainuch effects given the large customer base requirements. Moreesearch is needed is understand how web panels would impactustomers in these settings. Likewise, it would be interesting toxamine the web panel participants’ behavior in the absence ofnancial incentives as customers may differ in their sensitively

o such incentives. However, we note that most commercial webanels provide incentives to their members.

Second, because of the longitudinal field study methodology,e were not able to collect process measures to understand thenderlying psychological processes that led to these effects. For
nstance, an interesting question which deserves further probings about the relative contributions of increased attitude accessi-ility, increased positivity, and increased functional knowledgen producing the effects of survey participation frequency on
vas

iling 92 (2, 2016) 147–161 159

ustomer behaviors that we found. However, we note that lackf process measures is a common limitation in field studies, in theBE domain (e.g., Dholakia 2010; Dholakia and Morwitz 2002)

nd in other areas of marketing more broadly (e.g., Andersonnd Simester 2003).

Third, while our results have important implications, man-gers should exercise caution when extrapolating these resultsnto other situations and should carefully consider how closelyheir specific situation matches our empirical context. For exam-le, in hypothesizing a linear relationship between transactionehavior and number of surveys, we do so in the context ofeb panels. It is possible that web panels exhibit situational

diosyncrasies that make them different from traditional market-ng research. For example, customers in a web panel are fullyware in advance that they will be asked to answer surveys andhey have volunteered to do so. Thus customer wear-out might beifferent for web-panelists than for everyday customers. Addi-ionally, customers in web panels have an incentive to answer

ore surveys and thus may be more willing to participate thanrdinary consumers. Therefore, while we find a linear relation-hip between number of surveys and transactional behavior, its possible that customers participating in traditional marketingesearch, aside from web panels, may reach a point of diminish-ng returns due to wear-out or annoyance. Likewise, it is possiblehat a non-linear relationship or a tapering effect could occuromewhere beyond 27 surveys. Thus caution should be exercisedhen extrapolating beyond the range of the available surveys inur dataset.

Fourth, our data are from an online retailer. While this is a rel-tive strength of our dataset as we were able to comprehensivelyrack customers’ orders, care should be taken when extrapolatingur results to stores whose business relies heavily on brick-and-ortar outlets. Customers may have different considerationshen determining to shop online compared to brick-and-mortarutlets, such as the incurring time and travel costs when trans-cting with brick-and-mortar stores. Thus, it is unclear if theop-of-mind customer awareness that comes with web partici-ation will benefit brick-and-mortar stores the same as onlineetailers.

These limitations notwithstanding, our findings make sig-ificant contributions to the question behavior effect literature,nding practically and economically impactful effects of par-

icipating in proprietary web panels and discover customerifferences in susceptibility to such effects. For large retailers,roprietary customer web panels offer a compelling new wayo conduct survey-based research by circumventing many of themerging environmental challenges, and our research shows thathey have a positive economic impact on account of changes inustomer purchasing behaviors from participating in them.

Acknowledgements

We thank the firm and its market research company that pro-ided the data for this research, and Vikas Mittal, Grant Packard,nd Richard Gretz for providing useful comments on earlier ver-ions of this paper. We also express gratitude to Murali Mantrala

1 Reta

at

A

A

A

B

B

B

B

B

B

B

B

B

C

C

C

D

D

D

F

F

F

G

G

G

G

H

H

J

K

L

L

M

M

M

N

N

O

O

P

R

S

S


nd the three reviewers for their insightful feedback and supporthroughout the review process.

References

nderson, Eric T. and Duncan I. Simester (2003), “Effects of $9 Price Endingson Retail Sales: Evidence from Field Experiments,” Quantitative Marketingand Economics, 1 (1), 93–110.

tanassov, Julian (2013), “Do Hostile Takeovers Stifle Innovation? Evidencefrom the Antitakeover Legislation and Corporate Patenting,” Journal ofFinance, 68 (3), 1097–131.

utor, David H. (2003), “Outsourcing at Will: The Contribution of Unjust Dis-missal Doctrine to Growth of Employment Outsourcing,” Journal of LaborEconomics, 21 (1), 1–42.

aker, Reg, Stephen J. Blumberg, J. Michael Brick, Mick P. Couper, MelanieCourtright, J. Michael Dennis, Don Dillman, Martin R. Frankel, PhilipGarland, Robert M. Groves, Courtney Kennedy, Jon Krosnick and Paul J.Lavrakas (2010), “Research Synthesis AAPOR Report on Online Panels,”Public Opinion Quarterly, 74 (4), 711–81.

art, Yakov, Venkatesh Shankar, Fareena Sultan and Glen L. Urban (2005), “Arethe Drivers and Role of Online Trust the Same for All Web Sites and Con-sumers? A Large-Scale Exploratory Empirical Study,” Journal of Marketing,69 (4), 33–152.

erscheid, Ellen, Mark Snyder and Allen M. Omoto (1989), “The RelationshipCloseness Inventory: Assessing the Closeness of Interpersonal Relation-ships,” Journal of Personality and Social Psychology, 57 (5), 792–807.

ertrand, Marianne, Esther Duflo and Sendhil Mullainathan (2004), “How MuchShould We Trust Differences-in-Differences Estimates?,” Quarterly Journalof Economics, 119 (1), 249–75.

ertrand, Marianne and Sendhil Mullainathan (2003), “Enjoying the Quiet Life?Corporate Governance and Managerial Preferences,” Journal of PoliticalEconomy, 111 (5), 1043–75.

lundell, Richard, Alan Duncan and Costas Meghir (1998), “Estimating LaborSupply Responses Using Tax Reforms,” Econometrica, 66 (4), 827–61.

orle, Sharad, Utpal M. Dholakia, Siddharth S. Singh and Robert A. West-brook (2007), “The Impact of Survey Participation on Subsequent CustomerBehavior: An Empirical Investigation,” Marketing Science, 26 (5), 711–26.

ronnenberg, Bart J., Jean Pierre Dube and Carl F. Mela (2010), “Do DigitalVideo Recorders Influence Sales?,” Journal of Marketing Research, 47 (6),988–1010.

rüggen, Elisabeth and Utpal M. Dholakia (2009), “Determinants of Partici-pation and Response Effort in Web Panel Surveys,” Journal of InteractiveMarketing, 24 (3), 239–50.

allegaro, Mario and Charles DiSogra (2008), “Computing Response Metricsfor Online Panels,” Public Opinion Quarterly, 72 (5), 1008–32.

handon, Pierre, Vicki G. Morwitz and Werner J. Reinartz (2004), “The Short-and Long-term Effects of Measuring Intent to Repurchase,” Journal of Con-sumer Research, 31 (3), 566–72.

ommunispace (2015), “Case Studies,” (accessed January 23, 2015), [availableat http://www.communispace.com/Clients/CaseStudies/].

holakia, Utpal M. (2010), “A Critical Review of Question-Behavior EffectResearch,” Review of Marketing Research, 7, 147–99.

holakia, Utpal and Vicki Morwitz (2002), “The Scope and Persistence of Mere-Measurement Effects: Evidence from a Field Study of Customer SatisfactionMeasurement,” Journal of Consumer Research, 29 (September), 159–67.

holakia, Utpal, Vicki G. Morwitz and Robert A. Westbrook (2004), “Firm-Sponsored Satisfaction Surveys: Positivity Effects of Customer PurchaseBehavior?,” MSI Reports,, 4–121.

eldman, Jack M. and John G. Lynch (1988), “Self-Generated Validity andOther Effects of Measurement on Belief, Attitude, Intention, and Behavior,”Journal of Applied Psychology, 73 (3), 421–35.

uchs, Christoph and Martin Schreier (2011), “Customer Empowerment inNew Product Development,” Journal of Product Innovation Management,
28 (December), 17–32.
uchs, Christoph, Emanuela Prandelli and Martin Schreier (2010), “The Psy-chological Effects of Empowerment Strategies on Consumers’ ProductDemand,” Journal of Marketing, 74 (1), 65–79.

S

iling 92 (2, 2016) 147–161

ollwitzer, Peter M. and Gabriele Oettingen (2008), “The Question-BehaviorEffect from an Action Control Perspective,” Journal of Consumer Psychol-ogy, 18 (2), 107–10.

oritz, Anja (2004), “The Impact of Material Incentives on Response Quan-tity, Response Quality, Sample Construction, Survey Outcome, and Cost inOnline Access Panels,” International Journal of Market Research, 46 (3),327–45.

rewal, Dhruv and Michael Levy (2007), “Retailing Research: Past, Present,and Future,” Journal of Retailing, 83 (4), 447–64.

rewal, Dhruv, Michael Levy and V. Kumar (2009), “Customer ExperienceManagement in Retailing: An Organizing Framework,” Journal of Retailing,85 (1), 1–14.

amilton, Jamie, Marco Vriens and Misia Tramp (2008), “Building a ResearchCommunity,” Marketing Research, 20 (3), 16–22.

eckman, James J., Hidehiko Ichimura and Petra Todd (1998), “Matching asan Econometric Evaluation Estimator,” Review of Economic Studies, 65 (2),261–94.

aniszewski, Chris and Elise Chandon (2007), “Transfer-AppropriateProcessing, Response Fluency, and the Mere Measurement Effect,” Journalof Marketing Research, 44 (2), 309–23.

umar, V., Morris George and Joseph Pancras (2008), “Cross-Buying inRetailing: Drivers and Consequences,” Journal of Retailing, 84 (1),15–27.

evav, Jonathan and Gavan J. Fitzsimons (2006), “When Questions ChangeBehavior: The Role of Ease of Representation,” Psychological Science, 17(3), 207–13.

iu, Wendy and David Gal (2011), “Bringing us Together or Driving us Apart:The Effect of Soliciting Consumer Input on Consumers’ Propensity to Trans-act with an Organization,” Journal of Consumer Research, 38 (August),242–59.

anchanda, Puneet, Grant Packard and Adithya Pattabhiramaiah (2015),“Social Dollars: The Economic Impact of Customer Participation ina Firm-Sponsored Online Community,” Marketing Science, 34 (3),367–87.

ilardo, Robert M., Michael P. Johnson and Ted L. Huston (1983), “DevelopingClose Relationships: Changing Patterns of Interaction Between Pair Mem-bers and Social Networks,” Journal of Personality and Social Psychology,44 (5), 964–76.

orwitz, Vicki G. and Gavan J. Fitzsimons (2004), “The Mere MeasurementEffect: Why Does Measuring Intentions Change Actual Behavior?,” Journalof Consumer Psychology, 14 (1–2), 64–74.

ational Retail Federation (2013), (accessed July 13, 2013), [available athttp://www.nrf.com/modules.php?name=Pages&sp id=1140].

eslin, Scott A., Thomas P. Novak, Kenneth R. Baker and Donna L. Hoffman(2009), “An Optimal Contact Model for Maximizing Online Panel ResponseRates,” Management Science, 55 (5), 727–37.

’Donnell, Svenja (2013), “U.K. Retail Sales Rose in May on Furni-ture Demand, BRC Says,” (accessed July 13, 2013), [available athttp://www.bloomberg.com/news/2013-06-03/u-k-retail-sales-rose-in-may-on-furniture-demand-brc-says.html].

fir, Chezy, Itamar Simonson and Song-Oh Yoon (2009), “The Robust-ness of the Effects of Consumers’ Participation in Market Research:The Case of Service Quality Evaluations,” Journal of Marketing, 73 (6),105–14.

assenger (2015), “Clients,” (accessed January 13, 2015), [available athttp://www.thinkpassenger.com/clients].

inallo, Diego and Suman Basuroy (2009), “Does Advertising Spending Influ-ence Media Coverage of the Advertiser?,” Journal of Marketing, 73 (6),33–46.

chreier, Martin, Christoph Fuchs and Darren W. Dahl (2012), “The InnovationEffect of User Design: Exploring Consumers’ Innovation Perceptions ofFirms Selling Products Designed by Users,” Journal of Marketing, 76 (5),18–32.

irohi, Niren, Edward W. McLaughlin and Dick R. Wittink (1998), “A Model
of Consumer Perceptions and Store Loyalty Intentions for a SupermarketRetailer,” Journal of Retailing, 74 (2), 223–45.
prott, David E., Eric R. Spangenberg, Lauren G. Block, Gavan J. Fitzsimons,Vicki G. Morwitz and Patti Williams (2006), “The Question-Behavior Effect:

http://refhub.elsevier.com/S0022-4359(15)00097-4/sbref0005














































































































































































































































































































































































http://www.communispace.com/Clients/CaseStudies/


































































































































































































































































































































































































































































































































http://www.nrf.com/modules.php?name=Pages&sp_id=1140
























http://www.bloomberg.com/news/2013-06-03/u-k-retail-sales-rose-in-may-on-furniture-demand-brc-says.html

http://www.bloomberg.com/news/2013-06-03/u-k-retail-sales-rose-in-may-on-furniture-demand-brc-says.html




































http://www.thinkpassenger.com/clients






























































































Reta

S

S

V


What We Know and Where We Go from Here,” Social Influence, 1 (2),128–37.

tock, James H. and Mark W. Watson (2007), Introduction to Econometrics,Boston, MA: Pearson Education Inc.

ullivan, Elizabeth A. (2012), “10 Minutes With Bo Mattsson,” Marketing News,46 (15), 24–31.

W

iling 92 (2, 2016) 147–161 161

erhoef, Peter, Philip Hans Franses and Janny C. Hoekstra (2002), “The Effectof Relational Constructs on Customer Referrals and Number of Services
Purchased from a Multiservice Provider: Does Age of Relationship Matter?,”Journal of the Academy of Marketing Science, 30 (3), 202–16.
ooldridge, Jeffrey M. (2009), Introductory Econometrics: A ModernApproach, Mason, OH: South-Western Publishing.




































































































The Economic Benefits to Retailers from Customer ... · Journal of Retailing 92 (2, 2016) 147–161...

Documents

Transcript of The Economic Benefits to Retailers from Customer ... · Journal of Retailing 92 (2, 2016) 147–161...