Predicting Consumer Choice Using Supermarket Scanner Data:

Combining Parametric and Non-parametric Methods

Elena Enevaeneva@cs.cmu.edu

  April 20 2001

CALD Lab

Problem and Motivation

Profit optimization for storesKnow thy customerMicro-marketingPredicting Consumer ChoiceUse scanner dataPrevious research: profit increase Gross profit margin 4% to 10% Operating profit margin 33% to 83%

a classic business rule that is taught in every 100-level course

An under-utilized gold mine

Retail Store Scanner Data

Chilled Juice Category14 products2 years100 storesStore-level aggregationWeekly reports

Goal

Build an accurate predictor of consumer choice (that knows the customer). In-prices, out-quantities

Cate

gory

: C

hill

ed

Ora

ng

e Ju

ice

Price of Product 1

Price of Product 2

Price of Product 3

Price of Product 14

. . .

“I know your

customers”

PredictorPredictor

Quantity bought of Product 1

. . .

Quantity bought of Product 2

Quantity bought of Product 3

Quantity bought of Product 14

Previous Work on Retail Data

Traditionally – using parametric models (linear regression)Recently – using non-parametric models (neural networks)

NN outperforms LR in accuracy, although LR performs adequatelyNN are mistrusted

Our Niche

Advantage of LR: known functional form (linear in log space), extrapolation ability

Advantage of NN: flexibility, accuracy

extrapolation ability

acc

ura

cy

NN

new

LRTake Advantage: use the

assumed prior to bias the accurate learner

Higher level model from simpler models

Related Research In Other Fields

Patrice Simard et al. “Tangent Prop” Possible to directly learn the invariance of the data independently

from the “real” learning task

Michael Perrone “Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization” generalized ensemble method estimator

N

iiiGEM xff

1

)(

Combination Approaches

Train Separately, then Combine

Outputs as Inputs

Jumping Connections

Train separately, then combine

train a NN and a LR separately, and calculate the weighted average for the final prediction

NN

Input Prices

Output Quantities

. . .

. . .

LR

Input Prices

Output Quantities

. . .

. . .

Final Prediction

Outputs as Inputs

adding to a learner the prediction of the other learner (over the same 14 input prices) as an extra input

NN

Output Quantities

Input Prices

. . .

. . .

LR

. . .

Input Prices

. . .

Output Quantities

NN

Output Quantities

Input Prices

. . .

. . .

LR

. . .

Input Prices

. . .

Output Quantities

Jumping Connections

Combining two types of NN connections in one NNGives the effect of simulating a LR and NN all together

. . .

. . .

. . .

Results – RMS Errors

Linear RegressionNeural NetWeighted AverageJump ConnectionsOutputs as Inputs

0.1920.0750.0740.0720.070

Results - % Error in Predicted Q

Comparison with IO method: percent error in predicted output

0

1

2

3

4

5

6

IO JC WA NN LR

Summary

Combining NN and LR gives a more accurate and robust modelBetter in terms of understandability for the marketing communityLearns a better consumer modelImproves pricing strategies

Future Work

Include: demographics data promotional data competitor data

Apply Multitask LearningAnother data set with more density

End